Q: A data transfer between two AWS regions via the internet takes 10 seconds for a 100MB file (10 Mbps). You enable inter-region VPC peering, and the transfer completes in 0.1 seconds (10 Gbps). However, a large file transfer from within a VPC to an external S3 bucket in another region via the internet gateway bottlenecks at 100 Mbps. Why do VPC-to-VPC transfers saturate bandwidth while VPC-to-Internet transfers don't?
This illustrates the fundamental difference between AWS's internal backbone and the public internet.
Want to master this scenario in a live sandbox? KodeKloud's Istio Service Mesh & Advanced Kubernetes Networking Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "When an interviewer asks how I diagnose network connectivity, I follow an outside-in OSI model approach. The interviewer is testing: AWS network architecture, inter-region connectivity, bandwidth vs latency.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Initial Diagnostics & Root Cause Analysis
This illustrates the fundamental difference between AWS's internal backbone and the public internet.
- Traffic flows entirely over AWS's private fiber backbone (dedicated, optimized for high throughput).
- No congestion from public internet traffic.
- Low packet loss, consistent performance.
- All bandwidth is available (no ISP throttling or carrier limits).
- Traffic exits the VPC via the Internet Gateway (IGW), traversing public internet to reach S3's edge endpoints.
- S3 in other regions is accessed via public IP addresses (even though it's an AWS service).
- Bandwidth allocation: AWS typically allocates 100 Mbps per EC2 instance for internet egress (per AWS documentation, N1/T2/T3 instances). This is *per instance*, not per VPC.
- Once the instance exhausts its 100 Mbps allocation, throughput is capped, regardless of available physical bandwidth.
- Inter-region VPC peering: Traffic never leaves AWS's network. Uses dedicated peering connections optimized for high throughput.
- Internet traffic: Traverses public internet infrastructure (ISPs, CDNs, exchange points), all carrying millions of other users' traffic. AWS soft-caps per-instance internet throughput to prevent DoS.
Remediation & Permanent Safeguards
VPC-to-VPC Peering (10 Gbps achievable): VPC-to-Internet Gateway to S3 (100 Mbps bottleneck): Why this asymmetry? Solutions to increase VPC-to-S3 throughput: Can achieve ~1 Gbps for some use cases. This avoids the IGW bandwidth limitation and uses backbone; can achieve multi-Gbps. Recommended for high-throughput S3 (same region or cross-region): Use S3 VPC Gateway Endpoint (free, no data transfer charges) for all inter-region S3 access. It keeps traffic on the backbone and avoids the IGW bottleneck, achieving near-link-speed throughput.
VPC → S3 TA endpoint (nearest CloudFront edge) → S3 (via AWS backbone)
- Use AWS S3 Transfer Acceleration (leverages CloudFront edge locations):
- Create VPC Endpoint for S3 (Gateway endpoint):
- Increase instance size or use multiple instances:
- Larger instances (m5.2xlarge+) may have higher internet bandwidth allocations.
- Parallel transfers across multiple instances (each gets its own 100 Mbps quota).
- Dedicated Network Connection (Direct Connect):
- Expensive but guarantees dedicated bandwidth (1 Gbps, 10 Gbps, 100 Gbps).
- Bypasses public internet entirely; all traffic flows over AWS's private backbone.
- Traffic flows entirely over AWS's private fiber backbone (dedicated, optimized for high throughput).
- No congestion from public internet traffic.
- Low packet loss, consistent performance.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Relying on legacy bucket ACLs instead of modern S3 Block Public Access and IAM bucket policies when configuring Networking data transfer AWS recommends disabling ACLs (Bucket Owner Enforced) in favor of centralized declarative bucket policies and account-level Block Public Access controls.
-
Confusing SSE-S3 encryption with KMS customer-managed key (SSE-KMS) authorization policies when configuring Networking data transfer
Accessing SSE-KMS encrypted objects requires both
s3:GetObjectandkms:Decryptpermissions; neglecting KMS key policies causes silent 403 Forbidden errors. - Using sequential timestamps or predictable prefixes for high-throughput S3 read and write keys when configuring Networking data transfer Sequential key prefixes limit S3 horizontal partitioning; randomized hash prefixes distribute workloads across partitions to achieve 3,500 PUT and 5,500 GET requests/sec.
- Failing to audit S3 object delete markers when troubleshooting missing versioned objects when configuring Networking data transfer Issuing a DELETE on a versioned bucket only adds a delete marker; engineers must retrieve previous versions or purge markers to restore data.