Q: You have a fleet of EC2 instances behind an ALB. One instance keeps getting traffic even though it's unhealthy. What's wrong?
The ALB health check is likely misconfigured:
Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "In a previous role, our monitoring paged me for a similar incident across our AWS VPC infrastructure. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix."
Initial Diagnostics & Root Cause Analysis
The ALB health check is likely misconfigured:
- Health check path wrong — ALB checks
/but the app serves health at/health. Returns 404 → ALB marks it unhealthy → but wait, 404 might be in the success codes range. - Success codes misconfigured — if success codes include
404or5xx, unhealthy instances appear healthy. - Health check threshold —
UnhealthyThresholdmight be set very high, so the instance needs to fail many times before being marked unhealthy.
Remediation & Permanent Safeguards
Fix: Review ALB target group health check settings. Set correct path, success codes (200-299 only), and reasonable thresholds.
- Health check port/protocol wrong — checking wrong port.
- Security Group — ALB can't reach the instance's health check port → health checks time out → AWS treats timeout differently than explicit failure in some configs.
- Health check path wrong — ALB checks / but the app serves health at /health. Returns 404 → ALB ma...
- Success codes misconfigured — if success codes include 404 or 5xx, unhealthy instances appear hea...
- Health check threshold — UnhealthyThreshold might be set very high, so the instance needs to fail...
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Confusing ALB 502 Bad Gateway (target connection reset) with 504 Gateway Timeout (upstream response timeout) when configuring fleet instances behind 502 indicates the backend target closed the TCP connection prematurely or returned malformed HTTP, whereas 504 means the target failed to respond before idle timeout.
- Setting target deregistration delay to zero instead of allowing connection draining when configuring fleet instances behind Instant target deregistration severs in-flight HTTP requests during auto-scaling scale-in or rolling deployments, causing immediate user-facing drops.
-
Failing to query CloudWatch ELB metrics (
HTTPCode_ELB_5XXvsHTTPCode_Target_5XX) during load balancer outages when configuring fleet instances behind Comparing load balancer generated errors against target generated errors immediately isolates whether the issue lies in the proxy tier or application runtime. - Overlooking Security Group rules on backend EC2/ECS instances that block health check traffic from the ALB when configuring fleet instances behind Target instances must explicitly permit inbound HTTP/HTTPS traffic from the ALB security group on the configured container/application port.