Q: You have a fleet of EC2 instances behind an ALB. One instance keeps getting traffic even though it's unhealthy. What's wrong?
The ALB health check is likely misconfigured:
#AWS #EC2 & Compute #L2 #Cloud #Infrastructure #ALB
🎙️ Candidate Opening & Architectural Context
""In a previous role, our monitoring paged me for a similar incident across our AWS VPC infrastructure. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix.""
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Initial Diagnostics & Root Cause Analysis
The ALB health check is likely misconfigured:
- Health check path wrong — ALB checks
/but the app serves health at/health. Returns 404 → ALB marks it unhealthy → but wait, 404 might be in the success codes range. - Success codes misconfigured — if success codes include
404or5xx, unhealthy instances appear healthy. - Health check threshold —
UnhealthyThresholdmight be set very high, so the instance needs to fail many times before being marked unhealthy.
2️⃣
Remediation & Permanent Safeguards
Fix: Review ALB target group health check settings. Set correct path, success codes (200-299 only), and reasonable thresholds.
- Health check port/protocol wrong — checking wrong port.
- Security Group — ALB can't reach the instance's health check port → health checks time out → AWS treats timeout differently than explicit failure in some configs.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Health check path wrong — ALB checks / but the app serves health at /health. Returns 404 → ALB marks it unhealthy → but wait, 404 ."
⚡ 60-Second Elevator Pitch Talking Points
- Health check path wrong — ALB checks / but the app serves health at /health. Returns 404 → ALB ma...
- Success codes misconfigured — if success codes include 404 or 5xx, unhealthy instances appear hea...
- Health check threshold — UnhealthyThreshold might be set very high, so the instance needs to fail...
Advertisement