Q: A data transfer between two AWS regions via the internet takes 10 seconds for a 100MB file (10 Mbps). You enable inter-region VPC peering, and the transfer completes in 0.1 seconds (10 Gbps). However, a large file transfer from within a VPC to an external S3 bucket in another region via the internet gateway bottlenecks at 100 Mbps. Why do VPC-to-VPC transfers saturate bandwidth while VPC-to-Internet transfers don't?
This illustrates the fundamental difference between AWS's internal backbone and the public internet.
🛠️ Production Runbook & Step-by-Step Resolution
Initial Diagnostics & Root Cause Analysis
This illustrates the fundamental difference between AWS's internal backbone and the public internet.
- Traffic flows entirely over AWS's private fiber backbone (dedicated, optimized for high throughput).
- No congestion from public internet traffic.
- Low packet loss, consistent performance.
- All bandwidth is available (no ISP throttling or carrier limits).
- Traffic exits the VPC via the Internet Gateway (IGW), traversing public internet to reach S3's edge endpoints.
- S3 in other regions is accessed via public IP addresses (even though it's an AWS service).
- Bandwidth allocation: AWS typically allocates 100 Mbps per EC2 instance for internet egress (per AWS documentation, N1/T2/T3 instances). This is *per instance*, not per VPC.
- Once the instance exhausts its 100 Mbps allocation, throughput is capped, regardless of available physical bandwidth.
- Inter-region VPC peering: Traffic never leaves AWS's network. Uses dedicated peering connections optimized for high throughput.
- Internet traffic: Traverses public internet infrastructure (ISPs, CDNs, exchange points), all carrying millions of other users' traffic. AWS soft-caps per-instance internet throughput to prevent DoS.
Remediation & Permanent Safeguards
VPC-to-VPC Peering (10 Gbps achievable): VPC-to-Internet Gateway to S3 (100 Mbps bottleneck): Why this asymmetry? Solutions to increase VPC-to-S3 throughput: Can achieve ~1 Gbps for some use cases. This avoids the IGW bandwidth limitation and uses backbone; can achieve multi-Gbps. Recommended for high-throughput S3 (same region or cross-region): Use S3 VPC Gateway Endpoint (free, no data transfer charges) for all inter-region S3 access. It keeps traffic on the backbone and avoids the IGW bottleneck, achieving near-link-speed throughput.
VPC → S3 TA endpoint (nearest CloudFront edge) → S3 (via AWS backbone)
- Use AWS S3 Transfer Acceleration (leverages CloudFront edge locations):
- Create VPC Endpoint for S3 (Gateway endpoint):
- Increase instance size or use multiple instances:
- Larger instances (m5.2xlarge+) may have higher internet bandwidth allocations.
- Parallel transfers across multiple instances (each gets its own 100 Mbps quota).
- Dedicated Network Connection (Direct Connect):
- Expensive but guarantees dedicated bandwidth (1 Gbps, 10 Gbps, 100 Gbps).
- Bypasses public internet entirely; all traffic flows over AWS's private backbone.
- Traffic flows entirely over AWS's private fiber backbone (dedicated, optimized for high throughput).
- No congestion from public internet traffic.
- Low packet loss, consistent performance.