⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All AWS & Cloud Architecture Interview Questions Scenario 115 of 179 in AWS & Cloud Architecture
Staff SRE / Principal Architect [L3] AWS Cost & Architecture Staff SRE Scenario [L3]
🎯 Target Role / Context: AWS Scenario #118 (L3) · Cost & Architecture

Q: A fleet of 5,000 Lambda functions in a private VPC aggressively scrape data from the public internet. Randomly, hundreds of them begin crashing with bizarre `Connection Timed Out` networking errors, despite the internet destination being perfectly healthy. What AWS bottleneck is occurring?

This is classic SNAT (Source Network Address Translation) Port Exhaustion on the NAT Gateway.

#AWS #Cost & Architecture #L3 #Cloud #Infrastructure #VPC
🎙️ Candidate Opening & Architectural Context
""When an interviewer asks how I troubleshoot this in AWS, I frame it through my hands-on production experience. The interviewer is testing: NAT Gateway SNAT Port Exhaustion.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third.""
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: "When an interviewer asks how I troubleshoot this in AWS, I frame it through my hands-on production experience. The interviewer is testing: NAT Gateway SNAT Port Exhaustion.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."

1️⃣

Production Solution & Architecture

This is classic SNAT (Source Network Address Translation) Port Exhaustion on the NAT Gateway. A single AWS NAT Gateway utilizes a single public Elastic IP. TCP allows a theoretical maximum of ~65,000 ephemeral outbound ports per IP addressing a single destination. When 5,000 highly concurrent Lambda functions open thousands of individual API connections to the exact same external internet API simultaneously, the NAT Gateway completely runs out of ephemeral routing ports. It drops any new outbound connection attempts until old ones close. *Fix:* Heavily deploy multiple NAT Gateways across multiple public subnets and route traffic dynamically to distribute the SNAT allocation, or deploy dedicated NAT instances.

💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: This is classic SNAT (Source Network Address Translation) Port Exhaustion on the NAT Gateway.."
⚡ 60-Second Elevator Pitch Talking Points
  • Immediate Triage: This is classic SNAT (Source Network Address Translation) Port Exhaustion on the NAT Gateway.
  • Run targeted verification commands before modifying configuration.
  • Automate permanent guardrails (CI check, alerts, IaC policy) to prevent recurrence.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Purging compromised IAM user credentials immediately prior to querying CloudTrail forensic event logs when configuring fleet 5000 Lambda Deactivating a compromised key prevents abuse while preserving identity records for CloudTrail forensic queries to assess blast radius.
  2. Conflating local IAM Permissions Boundaries with organizational Service Control Policies (SCPs) when configuring fleet 5000 Lambda SCPs define maximum permission guards at the AWS Organization level across accounts, whereas permission boundaries limit delegate privileges within a single account.
  3. Deploying single-AZ cloud infrastructure for production workloads without multi-AZ failover resilience when configuring fleet 5000 Lambda Availability Zone hardware failures or localized network partitions cause complete service downtime when resources are not distributed across >=2 AZs.
  4. Jumping straight to SSH or instance restarts without reviewing CloudWatch metrics and AWS Systems Manager logs when configuring fleet 5000 Lambda Senior SREs triage control plane and hypervisor metrics via CloudWatch and SSM Session Manager before executing invasive host-level actions.
Advertisement
Want more AWS & Cloud Architecture scenarios?
Explore our complete collection of scenario-based AWS & Cloud Architecture interview runbooks.
Browse All AWS & Cloud Architecture Questions →