Q: Your Auto Scaling Group (ASG) keeps starting new EC2 instances, but they immediately fail the load balancer health checks and are subsequently terminated by the ASG in an infinite "thrashing" loop. How do you debug this?
If the instance is terminated immediately upon failing the health check, you never get a chance to SSH in and look at the logs.
#General DevOps #General DevOps — Scenario-Based Interview Questions #L2 #DevOps #SRE #Architecture
🎙️ Candidate Opening & Architectural Context
""When an interviewer asks about this scenario, I explain how we balanced incident response with long-term prevention. The interviewer is testing: ASG lifecycle hooks, scaling failure modes, graceful startup.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third.""
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Initial Diagnostics & Root Cause Analysis
If the instance is terminated immediately upon failing the health check, you never get a chance to SSH in and look at the logs.
- Suspend ASG Processes: I would temporarily suspend the
Terminateprocess on the ASG using the AWS Console or CLI. This allows the failing instance to remain alive so I can SSH in, check/var/log/cloud-init-output.logor the application logs, and find the syntax error or missing dependency crashing the boot script. - Check Grace Period: The application might naturally take 4 minutes to compile and start, but the ASG Health Check Grace Period might be set to 60 seconds. The ASG kills it prematurely because it hasn't finished booting. I would increase the Grace Period.
- Lifecycle Hooks: Use an ASG Lifecycle Hook (
autoscaling:EC2_INSTANCE_LAUNCHING) to place the instance in aPending:Waitstate while it boots, preventing the ALB from checking it until the instance explicitly signals it is fully ready.
2️⃣
Remediation & Permanent Safeguards
To debug:
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Suspend ASG Processes: I would temporarily suspend the Terminate process on the ASG using the AWS Console or CLI. This allows the ."
⚡ 60-Second Elevator Pitch Talking Points
- Suspend ASG Processes: I would temporarily suspend the Terminate process on the ASG using the AWS...
- Check Grace Period: The application might naturally take 4 minutes to compile and start, but the ...
- Lifecycle Hooks: Use an ASG Lifecycle Hook (autoscaling:EC2_INSTANCE_LAUNCHING) to place the inst...
Advertisement