Q: Your database is in a private subnet with a Network ACL (NACL) that explicitly allows port 3306 inbound from the application subnet (10.0.1.0/24). However, the DB connections are timing out. The Security Group allows 3306. What is wrong?
The problem is that Network ACLs are stateless.
Want to master this scenario in a live sandbox? KodeKloud's Istio Service Mesh & Advanced Kubernetes Networking Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "In our multi-VPC setup, services in private subnets ran into this exact routing obstacle. The interviewer is testing: Ephemeral ports, stateful vs. stateless firewalls.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Production Solution & Architecture
The problem is that Network ACLs are stateless. Security Groups are stateful (if you allow an inbound request, the outbound response is automatically allowed). Because NACLs are stateless, returning traffic is blocked unless explicitly permitted. When the application server hits the DB on port 3306, the database must reply to the application server's random Ephemeral Port (usually ranging from 1024-65535, typical Linux is 32768-60999). You must add an outbound rule on the database subnet's NACL to allow TCP traffic across the ephemeral port range back to the application subnet 10.0.1.0/24.
- Immediate Triage: The problem is that Network ACLs are stateless.
- Run targeted verification commands before modifying configuration.
- Automate permanent guardrails (CI check, alerts, IaC policy) to prevent recurrence.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Treating stateless Network ACLs as stateful firewalls without specifying outbound ephemeral return ports when configuring Networking database private Security Groups track connection state automatically, whereas stateless Network ACLs require explicit ephemeral port rules (1024-65535) for return traffic.
- Opening short-lived non-pooled TCP client connections, exhausting ephemeral ports in TIME_WAIT state when configuring Networking database private Rapidly opening and closing short-lived TCP sockets without connection pooling exhausts ephemeral ports, causing connection refusal errors.
- Directing load balancer health checks to heavy computational endpoints rather than dedicated health routes when configuring Networking database private Deep health check paths amplify database load during traffic spikes and risk cascading eviction of healthy application nodes.
- Triggering BGP route dampening by rapidly announcing and withdrawing multihomed network prefixes when configuring Networking database private Rapidly flapping BGP route announcements triggers upstream ISP route dampening, effectively blackholing network traffic for hours.