Q: You enable VPC Flow Logs on a production VPC, dumping 100GB per day of accept/reject traffic to S3. A developer is having connectivity issues, but analyzing raw logs is impossible. What queries do you run to isolate the problematic traffic pattern?
VPC Flow Logs capture every packet at the ENI (Elastic Network Interface) level. Each log entry includes source IP, destination IP, port, action (ACCEPT/REJECT), and protocol.
Want to master this scenario in a live sandbox? KodeKloud's Istio Service Mesh & Advanced Kubernetes Networking Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "Networking issues can paralyze distributed applications. In our hybrid cloud architecture, we traced this packet path. The interviewer is testing: VPC Flow Logs interpretation, log analysis, network troubleshooting.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Initial Diagnostics & Root Cause Analysis
VPC Flow Logs capture every packet at the ENI (Elastic Network Interface) level. Each log entry includes source IP, destination IP, port, action (ACCEPT/REJECT), and protocol.
- More REJECT than ACCEPT on a port: NACL or Security Group rules are asymmetric (outbound allowed but inbound blocked).
- ACCEPT logged but application still fails: Application isn't listening, or OS firewall is blocking (check target security group egress rules).
- No logs at all for a destination: Traffic never reached the VPC (routing issue upstream).
Remediation & Permanent Safeguards
Quick queries (using Athena/S3 SQL): Find all rejected traffic to a specific destination: This reveals: Is traffic being dropped at the NACL or Security Group level? From which source IPs? Check if the destination itself is rejecting or the network layer is: Root cause scenarios:
SELECT srcaddr, dstaddr, dstport, protocol, COUNT(*) as attempts
FROM vpc_flow_logs
WHERE dstaddr = '10.0.2.50' -- The destination with issues
AND action = 'REJECT'
AND day >= '2025-01-15'
GROUP BY srcaddr, dstaddr, dstport, protocol
ORDER BY attempts DESC
- More REJECT than ACCEPT on a port: NACL or Security Group rules are asymmetric (outbound allowed ...
- ACCEPT logged but application still fails: Application isn't listening, or OS firewall is blockin...
- No logs at all for a destination: Traffic never reached the VPC (routing issue upstream).
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Relying on legacy bucket ACLs instead of modern S3 Block Public Access and IAM bucket policies when configuring Networking enable Flow AWS recommends disabling ACLs (Bucket Owner Enforced) in favor of centralized declarative bucket policies and account-level Block Public Access controls.
-
Confusing SSE-S3 encryption with KMS customer-managed key (SSE-KMS) authorization policies when configuring Networking enable Flow
Accessing SSE-KMS encrypted objects requires both
s3:GetObjectandkms:Decryptpermissions; neglecting KMS key policies causes silent 403 Forbidden errors. - Using sequential timestamps or predictable prefixes for high-throughput S3 read and write keys when configuring Networking enable Flow Sequential key prefixes limit S3 horizontal partitioning; randomized hash prefixes distribute workloads across partitions to achieve 3,500 PUT and 5,500 GET requests/sec.
- Failing to audit S3 object delete markers when troubleshooting missing versioned objects when configuring Networking enable Flow Issuing a DELETE on a versioned bucket only adds a delete marker; engineers must retrieve previous versions or purge markers to restore data.