Q: We have a BGP VPN connection established over IPSec from our data center to AWS. The tunnel is "UP", but large file transfers keep freezing or failing halfway through, while small pings and SSH commands work fine. What is happening?
This is absolutely an MTU (Maximum Transmission Unit) mismatch issue.
#Networking #Networking #L3 #VPC #DNS #Security
🎙️ Candidate Opening & Architectural Context
""In our multi-VPC setup, services in private subnets ran into this exact routing obstacle. The interviewer is testing: MTU (Maximum Transmission Unit), MSS, Path MTU Discovery, IPsec header overhead.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third.""
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Initial Diagnostics & Root Cause Analysis
This is absolutely an MTU (Maximum Transmission Unit) mismatch issue.
- Enable
TCP MSS Clampingon the VPN router (modifying the TCP handshake to force a lower MSS, e.g., 1350). - Allow ICMP type 3 code 4 (Fragmentation Needed) on all firewalls.
- Lower the MTU manually on the host interfaces.
2️⃣
Remediation & Permanent Safeguards
Standard ethernet MTU is 1500 bytes. However, IPsec VPN tunnels add encryption headers (ESP, tunnel mode) which consume ~50-80 bytes. If an application tries to send a full 1500-byte packet with the "Don't Fragment" (DF) bit set, the VPN router drops it because it exceeds the tunnel's inner MTU, and sends an ICMP "Fragmentation Needed" message back. However, if firewalls along the path are blocking these ICMP messages (breaking Path MTU Discovery), the application never slows down its packet size. Known as a "PMTUD Blackhole". To fix:
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Enable TCP MSS Clamping on the VPN router (modifying the TCP handshake to force a lower MSS, e.g., 1350).."
⚡ 60-Second Elevator Pitch Talking Points
- Enable TCP MSS Clamping on the VPN router (modifying the TCP handshake to force a lower MSS, e.g....
- Allow ICMP type 3 code 4 (Fragmentation Needed) on all firewalls.
- Lower the MTU manually on the host interfaces.
Advertisement