⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Networking & Cloud DNS Interview Questions Scenario 42 of 55 in Networking & Cloud DNS
Staff SRE / Principal Architect [L3] Networking Staff SRE Scenario [L3]
🎯 Target Role / Context: Networking Scenario #43 (L3) · Networking

Q: Your latency between London and Tokyo (transcontinental WAN link) is high on a single large file transfer (scp, 50GB file). Speedtest shows 10 Gbps available, but SCP maxes out at 150 Mbps. The link is 99% idle. Why is TCP not filling the available bandwidth, and what is the root cause?

This is a classic TCP Window Size limitation over high-latency links. TCP's congestion window grows slowly, and it's fundamentally designed for Local Area Networks (LAN latency ~1ms), not intercontinental links (~150ms RTT).

#Networking #L3 #VPC #DNS #Security
🎙️ Candidate Opening & Architectural Context
""In our multi-VPC setup, services in private subnets ran into this exact routing obstacle. The interviewer is testing: TCP Window Scaling, RTT impact, long-distance performance tuning.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third.""
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's Istio Service Mesh & Advanced Kubernetes Networking Course covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: "In our multi-VPC setup, services in private subnets ran into this exact routing obstacle. The interviewer is testing: TCP Window Scaling, RTT impact, long-distance performance tuning.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."

1️⃣

Initial Diagnostics & Root Cause Analysis

This is a classic TCP Window Size limitation over high-latency links. TCP's congestion window grows slowly, and it's fundamentally designed for Local Area Networks (LAN latency ~1ms), not intercontinental links (~150ms RTT).

  • SCP sends ~2MB per window, then waits 150ms for ACK before sending more.
  • In 150ms of waiting, the pipe sits idle. The bandwidth is *available*, but TCP isn't using it due to the window limitation.
  • Increase TCP Window Size (Quick fix):
  • Use a faster transfer tool (Better):
  • bbcp (Big Brother Copy): Custom protocol that opens multiple parallel TCP streams and handles congestion better over WAN.
2️⃣

Remediation & Permanent Safeguards

Root Cause Math: TCP's maximum throughput is: Throughput = (TCP_Window_Size / RTT) London-Tokyo RTT ≈ 150ms (150,000 microseconds). Default TCP window on Linux: 64KB. Max throughput = 64KB / 0.15s = 3.4 Mbps Why SCP only achieves 150 Mbps instead of 10 Gbps? Solutions: This allows the window to scale up dynamically (RFC 7323 TCP Window Scaling). New calculation: Max = 64MB / 0.15s = 3.4 Gbps (closer to the available 10 Gbps). Lesson: WAN performance is fundamentally about RTT and window size, not raw bandwidth. A 10 Gbps link with 150ms latency can transfer ~3.4 Gbps max unless you increase the window.

sysctl -w net.ipv4.tcp_rmem='4096 87380 67108864'  # 64MB max
   sysctl -w net.ipv4.tcp_wmem='4096 65536 67108864'  # 64MB max
  • perfsonar: Network tuning tool that automatically adjusts window sizes and buffer for your specific latency.
  • rclone: Transfer tool designed for cloud workflows, handles retries and multi-part uploads.
  • Enable TCP Fast Open + SACK (Modern):
  • UDP-based alternatives: For non-reliable-delivery systems, use QUIC (HTTP/3) or custom UDP, which don't have the ACK-window bottleneck.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: SCP sends ~2MB per window, then waits 150ms for ACK before sending more.."
⚡ 60-Second Elevator Pitch Talking Points
  • SCP sends ~2MB per window, then waits 150ms for ACK before sending more.
  • In 150ms of waiting, the pipe sits idle. The bandwidth is *available*, but TCP isn't using it due...
  • Increase TCP Window Size (Quick fix):
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Treating stateless Network ACLs as stateful firewalls without specifying outbound ephemeral return ports when configuring Networking latency London Security Groups track connection state automatically, whereas stateless Network ACLs require explicit ephemeral port rules (1024-65535) for return traffic.
  2. Opening short-lived non-pooled TCP client connections, exhausting ephemeral ports in TIME_WAIT state when configuring Networking latency London Rapidly opening and closing short-lived TCP sockets without connection pooling exhausts ephemeral ports, causing connection refusal errors.
  3. Directing load balancer health checks to heavy computational endpoints rather than dedicated health routes when configuring Networking latency London Deep health check paths amplify database load during traffic spikes and risk cascading eviction of healthy application nodes.
  4. Triggering BGP route dampening by rapidly announcing and withdrawing multihomed network prefixes when configuring Networking latency London Rapidly flapping BGP route announcements triggers upstream ISP route dampening, effectively blackholing network traffic for hours.
Advertisement
Want more Networking & Cloud DNS scenarios?
Explore our complete collection of scenario-based Networking & Cloud DNS interview runbooks.
Browse All Networking & Cloud DNS Questions →