Senior DevOps / SRE AWS VPC & Networking Networking & Security

Q: Application works internally but not from the internet — how would you troubleshoot?

Outside-in OSI network layer troubleshooting model for resolving applications reachable internally on private VPC IPs but inaccessible over the public internet.

#AWS #VPC #Route 53 #Internet Gateway #Security Groups #NACL

🎙️ Candidate Opening & Architectural Context

"Since the application works internally, the compute instance and software service are healthy. The failure is strictly along the ingress network path between the public internet and the AWS VPC."

🛠️ Production Runbook & Step-by-Step Resolution

1️⃣

DNS & IP Resolution (Outside-in Step 1)

Verify public DNS mapping from an external workstation:

Run dig +short app.example.com and nslookup app.example.com.
Does it resolve to a public IP or ALB CNAME? (Common mistake: Route 53 public hosted zone was not updated, or only a private hosted zone exists).
Is the resolved public IP reachable via traceroute or curl -Iv https://app.example.com?

2️⃣

Entry Point Topology & Subnet Routing (Step 2)

Verify the Internet Gateway and subnet routing tables:

Is the ALB / Gateway in a Public Subnet? A public subnet must have a Route Table entry pointing 0.0.0.0/0 → igw-xxxx (Internet Gateway). If an ALB is mistakenly placed in private subnets, internet packets cannot reach it.
Public IPv4 Addressing: If accessing an EC2 instance directly, does it have an Elastic IP (EIP) or auto-assigned public IP?
NAT Gateway Confusion: NAT Gateways only enable outbound egress from private subnets to internet; they DO NOT accept inbound ingress from the internet.

3️⃣

Firewalls: Security Groups & NACLs (Step 3)

Inspect AWS stateful and stateless firewall layers:

ALB Security Group: Must allow inbound ports 80/443 from 0.0.0.0/0 (Internet).
Backend EC2 Security Group: Must allow traffic from the ALB's Security Group ID (chained SG reference).
Network ACLs (NACL): Must have an allow rule for inbound 80/443, AND allow outbound ephemeral return traffic on ports 1024–65535.

4️⃣

AWS WAF & Target Group Binding (Step 4)

Check perimeter protection and target routing:

AWS WAF: Check if an attached Web ACL is blocking requests due to IP rate limits, geographic blocking, or SQLi false positives (look for 403 Forbidden in WAF metrics).
Target Group Binding: Ensure the target group has healthy registered instances and correct port mappings.

💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)

"Trace outside-in: DNS -> Internet Gateway & Route Table -> Public Subnet ALB -> Security Group (80/443 from 0.0.0.0/0) -> NACLs (ephemeral return) -> WAF. Internal working proves compute is fine; focus purely on the AWS ingress pipeline."

⚡ 60-Second Elevator Pitch Talking Points

Check DNS: dig app.example.com to verify public resolution to ALB CNAME / Elastic IP.
Check Route Table: ensure ALB is in public subnets with route 0.0.0.0/0 -> Internet Gateway (IGW).
Check Security Groups: ALB SG must allow 80/443 from 0.0.0.0/0; instance SG must allow traffic from ALB SG.
Check NACLs: ensure stateless NACLs permit both inbound 80/443 and outbound ephemeral ports (1024-65535).
Check AWS WAF: inspect blocked requests in WAF console for geo-blocking or rate-limit blocks.
Use AWS VPC Reachability Analyzer to mathematically prove the network path between IGW and instance.