Q: An EC2 instance in an Auto Scaling Group keeps being terminated and replaced. The new instance starts, becomes healthy, then gets terminated again in a cycle. What's happening?
This sounds like a lifecycle hook or health check issue:
Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "AWS reliability requires differentiating between AWS control plane limits and host-level resource exhaustion. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix."
Initial Diagnostics & Root Cause Analysis
This sounds like a lifecycle hook or health check issue:
- Application actually is unhealthy — the new instance really is failing. Check the health check endpoint. Maybe there's a deployment bug.
- ALB health check failing immediately — instance isn't ready to serve traffic before health checks kick in. The ASG uses ALB health checks and terminates the instance before the app is fully started.
- Lifecycle hook stuck — a lifecycle hook (
autoscaling:EC2_INSTANCE_LAUNCHING) is not completing. Instance is inPending:Waitstate but something is killing it.
Remediation & Permanent Safeguards
Check ASG activity logs in the console — it will say exactly why each instance was terminated.
- Scale-in protection not set — instance is being terminated by scale-in event despite looking fine.
- EC2 instance reachability check failing — hardware issue at the hypervisor level. Check AWS console system status checks.
- Application actually is unhealthy — the new instance really is failing. Check the health check en...
- ALB health check failing immediately — instance isn't ready to serve traffic before health checks...
- Lifecycle hook stuck — a lifecycle hook (autoscaling:EC2_INSTANCE_LAUNCHING) is not completing. I...
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Confusing ALB 502 Bad Gateway (target connection reset) with 504 Gateway Timeout (upstream response timeout) when configuring instance Auto Scaling 502 indicates the backend target closed the TCP connection prematurely or returned malformed HTTP, whereas 504 means the target failed to respond before idle timeout.
- Setting target deregistration delay to zero instead of allowing connection draining when configuring instance Auto Scaling Instant target deregistration severs in-flight HTTP requests during auto-scaling scale-in or rolling deployments, causing immediate user-facing drops.
-
Failing to query CloudWatch ELB metrics (
HTTPCode_ELB_5XXvsHTTPCode_Target_5XX) during load balancer outages when configuring instance Auto Scaling Comparing load balancer generated errors against target generated errors immediately isolates whether the issue lies in the proxy tier or application runtime. - Overlooking Security Group rules on backend EC2/ECS instances that block health check traffic from the ALB when configuring instance Auto Scaling Target instances must explicitly permit inbound HTTP/HTTPS traffic from the ALB security group on the configured container/application port.