⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All AWS & Cloud Architecture Interview Questions Scenario 3 of 179 in AWS & Cloud Architecture
Senior DevOps / SRE AWS Load Balancing High-Severity Incident
🎯 Target Role / Context: Production SRE Incident Analysis

Q: ALB starts returning 5xx errors — how would you identify the root cause?

Production runbook for isolating Application Load Balancer (ALB) 5xx errors: distinguishing ELB-generated vs Target-generated codes, debugging 502/503/504, and querying access logs with Athena.

#AWS #ALB #CloudWatch #Target Groups #HTTP 502 #HTTP 504
🎙️ Candidate Opening & Architectural Context
"When an ALB starts throwing 5xx errors, my immediate first goal is to differentiate between errors generated by the ALB itself versus errors generated by backend application targets."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: When an ALB starts throwing 5xx errors, my immediate first goal is to differentiate between errors generated by the ALB itself versus errors generated by backend application targets.

1️⃣

CloudWatch Metrics Differentiation (ELB vs Target)

Look at CloudWatch metrics for the load balancer immediately:

  • HTTPCode_ELB_5XX_Count: The ALB itself generated the error before receiving a response from targets (502 Bad Gateway, 503 Service Unavailable, 504 Gateway Timeout).
  • HTTPCode_Target_5XX_Count: The backend application generated a 5xx response (e.g. unhandled 500 internal server error or database crash), and the ALB simply forwarded it to the client.
  • HealthyHostCount / UnHealthyHostCount: Check if backend targets are failing health checks.
2️⃣

Diagnose Specific 5xx Error Codes

Apply targeted root cause analysis based on status code:

  • HTTP 502 (Bad Gateway): Target closed the TCP connection before sending full HTTP headers. Classic cause: Keep-Alive timeout mismatch (backend web server keep-alive must be set GREATER than ALB idle timeout, e.g. backend 65s vs ALB 60s).
  • HTTP 503 (Service Unavailable): No healthy targets available in the Target Group (all targets failed health checks), or ALB capacity exceeded during extreme traffic spikes.
  • HTTP 504 (Gateway Timeout): Target took longer to respond than the ALB idle timeout. Check for slow database queries, thread pool exhaustion, or locked worker processes.
  • HTTP 500 (Target Generated): Application code threw an uncaught exception, or database/Redis connection failed.
Advertisement
3️⃣

Query ALB Access Logs with Amazon Athena

Analyze ALB S3 access logs to extract exact failing requests, targets, and response times:

  • Check elb_status_code vs target_status_code.
  • Check target_processing_time: If high (>30s), target application is hanging.
  • Identify offending request paths, client IPs, or specific unhealthy backend target IP addresses.
4️⃣

Remediate and Safeguard

Immediate mitigation and permanent safeguards:

  • If 503 (unhealthy targets): Inspect target health check path (e.g. /healthz) in Target Group. Are targets returning 404 or failing DB ping?
  • If 502: Align keep-alive timeout: set Nginx/Node.js/Gunicorn keep-alive to 65s+ and ALB timeout to 60s.
  • If 504: Temporarily bump ALB idle timeout to relieve immediate pressure while developers optimize slow DB queries.
  • Configure CloudWatch Alarms on HTTPCode_ELB_5XX_Count > 10 with SNS to PagerDuty.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Always split ELB 5xx from Target 5xx in CloudWatch first. If it's ELB 502, check Keep-Alive timeout mismatches; if 503, check Target Group healthy host count; if 504, check target processing latency."
⚡ 60-Second Elevator Pitch Talking Points
  • Check CloudWatch: HTTPCode_ELB_5XX (ALB fault) vs HTTPCode_Target_5XX (app fault).
  • If Target 5XX: App is throwing 500s; inspect app logs and database connectivity.
  • If ELB 502: Target closed TCP prematurely; fix Keep-Alive timeout (backend keep-alive must exceed ALB 60s).
  • If ELB 503: Zero healthy targets in Target Group; verify health check path (/healthz) and target capacity.
  • If ELB 504: Gateway timeout; target query took >60s. Check target_processing_time and database locks.
  • Query ALB Access Logs via Athena to isolate failing URLs, client IPs, and specific target instance IDs.
🎬 60-Second Video Walkthrough
Watch the Animated Breakdown & Practice Live

Mastered this procedure? Subscribe to @naveedkumbhar01 on YouTube for daily 60-second SRE incident runbooks, Kubernetes triage shortcuts, and high-impact interview elevator pitches.

🔔 Subscribe on YouTube (Free) ▶ Watch Short (60s) ↗
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Confusing ALB 502 Bad Gateway (target connection reset) with 504 Gateway Timeout (upstream response timeout) when configuring Starts Returning Errors 502 indicates the backend target closed the TCP connection prematurely or returned malformed HTTP, whereas 504 means the target failed to respond before idle timeout.
  2. Setting target deregistration delay to zero instead of allowing connection draining when configuring Starts Returning Errors Instant target deregistration severs in-flight HTTP requests during auto-scaling scale-in or rolling deployments, causing immediate user-facing drops.
  3. Failing to query CloudWatch ELB metrics (HTTPCode_ELB_5XX vs HTTPCode_Target_5XX) during load balancer outages when configuring Starts Returning Errors Comparing load balancer generated errors against target generated errors immediately isolates whether the issue lies in the proxy tier or application runtime.
  4. Overlooking Security Group rules on backend EC2/ECS instances that block health check traffic from the ALB when configuring Starts Returning Errors Target instances must explicitly permit inbound HTTP/HTTPS traffic from the ALB security group on the configured container/application port.
Advertisement
Want more AWS & Cloud Architecture scenarios?
Explore our complete collection of scenario-based AWS & Cloud Architecture interview runbooks.
Browse All AWS & Cloud Architecture Questions →