Q: Your Auto Scaling Group (ASG) keeps starting new EC2 instances, but they immediately fail the load balancer health checks and are subsequently terminated by the ASG in an infinite "thrashing" loop. How do you debug this?
If the instance is terminated immediately upon failing the health check, you never get a chance to SSH in and look at the logs.
Want to master this scenario in a live sandbox? The Linux Foundation's Certified System Administrator (LFCS) & SRE Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "When an interviewer asks about this scenario, I explain how we balanced incident response with long-term prevention. The interviewer is testing: ASG lifecycle hooks, scaling failure modes, graceful startup.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Initial Diagnostics & Root Cause Analysis
If the instance is terminated immediately upon failing the health check, you never get a chance to SSH in and look at the logs.
- Suspend ASG Processes: I would temporarily suspend the
Terminateprocess on the ASG using the AWS Console or CLI. This allows the failing instance to remain alive so I can SSH in, check/var/log/cloud-init-output.logor the application logs, and find the syntax error or missing dependency crashing the boot script. - Check Grace Period: The application might naturally take 4 minutes to compile and start, but the ASG Health Check Grace Period might be set to 60 seconds. The ASG kills it prematurely because it hasn't finished booting. I would increase the Grace Period.
- Lifecycle Hooks: Use an ASG Lifecycle Hook (
autoscaling:EC2_INSTANCE_LAUNCHING) to place the instance in aPending:Waitstate while it boots, preventing the ALB from checking it until the instance explicitly signals it is fully ready.
Remediation & Permanent Safeguards
To debug:
- Suspend ASG Processes: I would temporarily suspend the Terminate process on the ASG using the AWS...
- Check Grace Period: The application might naturally take 4 minutes to compile and start, but the ...
- Lifecycle Hooks: Use an ASG Lifecycle Hook (autoscaling:EC2_INSTANCE_LAUNCHING) to place the inst...
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Confusing ALB 502 Bad Gateway (target connection reset) with 504 Gateway Timeout (upstream response timeout) when configuring General Auto Scaling 502 indicates the backend target closed the TCP connection prematurely or returned malformed HTTP, whereas 504 means the target failed to respond before idle timeout.
- Setting target deregistration delay to zero instead of allowing connection draining when configuring General Auto Scaling Instant target deregistration severs in-flight HTTP requests during auto-scaling scale-in or rolling deployments, causing immediate user-facing drops.
-
Failing to query CloudWatch ELB metrics (
HTTPCode_ELB_5XXvsHTTPCode_Target_5XX) during load balancer outages when configuring General Auto Scaling Comparing load balancer generated errors against target generated errors immediately isolates whether the issue lies in the proxy tier or application runtime. - Overlooking Security Group rules on backend EC2/ECS instances that block health check traffic from the ALB when configuring General Auto Scaling Target instances must explicitly permit inbound HTTP/HTTPS traffic from the ALB security group on the configured container/application port.