Q: ALB starts returning 5xx errors — how would you identify the root cause?
Production runbook for isolating Application Load Balancer (ALB) 5xx errors: distinguishing ELB-generated vs Target-generated codes, debugging 502/503/504, and querying access logs with Athena.
ALB Returning 5XX Errors? Split ELB vs Target FIRST
Watch Naveed break down how to troubleshoot AWS Application Load Balancer (ALB) 5xx errors—differentiating HTTPCode_ELB_5XX vs HTTPCode_Target_5XX, fixing 502 keep-alive timeouts, and debugging 503/504 outages in 60 seconds.
- CloudWatch first: Differentiate HTTPCode_ELB_5XX (ALB) vs HTTPCode_Target_5XX (backend app)
- Fix HTTP 502 Bad Gateway: Check Keep-Alive timeout (backend keep-alive > ALB 60s idle timeout)
- Debug HTTP 503 & 504: Unhealthy target group host count vs target processing latency
- Athena SQL queries on access logs to isolate client IPs, status codes, and failing target instances
Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: When an ALB starts throwing 5xx errors, my immediate first goal is to differentiate between errors generated by the ALB itself versus errors generated by backend application targets.
CloudWatch Metrics Differentiation (ELB vs Target)
Look at CloudWatch metrics for the load balancer immediately:
HTTPCode_ELB_5XX_Count: The ALB itself generated the error before receiving a response from targets (502 Bad Gateway, 503 Service Unavailable, 504 Gateway Timeout).HTTPCode_Target_5XX_Count: The backend application generated a 5xx response (e.g. unhandled 500 internal server error or database crash), and the ALB simply forwarded it to the client.HealthyHostCount/UnHealthyHostCount: Check if backend targets are failing health checks.
Diagnose Specific 5xx Error Codes
Apply targeted root cause analysis based on status code:
- HTTP 502 (Bad Gateway): Target closed the TCP connection before sending full HTTP headers. Classic cause: Keep-Alive timeout mismatch (backend web server keep-alive must be set GREATER than ALB idle timeout, e.g. backend 65s vs ALB 60s).
- HTTP 503 (Service Unavailable): No healthy targets available in the Target Group (all targets failed health checks), or ALB capacity exceeded during extreme traffic spikes.
- HTTP 504 (Gateway Timeout): Target took longer to respond than the ALB idle timeout. Check for slow database queries, thread pool exhaustion, or locked worker processes.
- HTTP 500 (Target Generated): Application code threw an uncaught exception, or database/Redis connection failed.
Query ALB Access Logs with Amazon Athena
Analyze ALB S3 access logs to extract exact failing requests, targets, and response times:
- Check
elb_status_codevstarget_status_code. - Check
target_processing_time: If high (>30s), target application is hanging. - Identify offending request paths, client IPs, or specific unhealthy backend target IP addresses.
Remediate and Safeguard
Immediate mitigation and permanent safeguards:
- If 503 (unhealthy targets): Inspect target health check path (e.g.
/healthz) in Target Group. Are targets returning 404 or failing DB ping? - If 502: Align keep-alive timeout: set Nginx/Node.js/Gunicorn keep-alive to 65s+ and ALB timeout to 60s.
- If 504: Temporarily bump ALB idle timeout to relieve immediate pressure while developers optimize slow DB queries.
- Configure CloudWatch Alarms on
HTTPCode_ELB_5XX_Count > 10with SNS to PagerDuty.
- Check CloudWatch: HTTPCode_ELB_5XX (ALB fault) vs HTTPCode_Target_5XX (app fault).
- If Target 5XX: App is throwing 500s; inspect app logs and database connectivity.
- If ELB 502: Target closed TCP prematurely; fix Keep-Alive timeout (backend keep-alive must exceed ALB 60s).
- If ELB 503: Zero healthy targets in Target Group; verify health check path (/healthz) and target capacity.
- If ELB 504: Gateway timeout; target query took >60s. Check target_processing_time and database locks.
- Query ALB Access Logs via Athena to isolate failing URLs, client IPs, and specific target instance IDs.
Mastered this procedure? Subscribe to @naveedkumbhar01 on YouTube for daily 60-second SRE incident runbooks, Kubernetes triage shortcuts, and high-impact interview elevator pitches.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Confusing ALB 502 Bad Gateway (target connection reset) with 504 Gateway Timeout (upstream response timeout) when configuring Starts Returning Errors 502 indicates the backend target closed the TCP connection prematurely or returned malformed HTTP, whereas 504 means the target failed to respond before idle timeout.
- Setting target deregistration delay to zero instead of allowing connection draining when configuring Starts Returning Errors Instant target deregistration severs in-flight HTTP requests during auto-scaling scale-in or rolling deployments, causing immediate user-facing drops.
-
Failing to query CloudWatch ELB metrics (
HTTPCode_ELB_5XXvsHTTPCode_Target_5XX) during load balancer outages when configuring Starts Returning Errors Comparing load balancer generated errors against target generated errors immediately isolates whether the issue lies in the proxy tier or application runtime. - Overlooking Security Group rules on backend EC2/ECS instances that block health check traffic from the ALB when configuring Starts Returning Errors Target instances must explicitly permit inbound HTTP/HTTPS traffic from the ALB security group on the configured container/application port.