⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All AWS & Cloud Architecture Interview Questions Scenario 11 of 179 in AWS & Cloud Architecture
Staff SRE / Principal Architect [L3] AWS EC2 & Compute Staff SRE Scenario [L3]
🎯 Target Role / Context: AWS Scenario #4 (L3) · EC2 & Compute

Q: An EC2 instance in an Auto Scaling Group keeps being terminated and replaced. The new instance starts, becomes healthy, then gets terminated again in a cycle. What's happening?

This sounds like a lifecycle hook or health check issue:

#AWS #EC2 & Compute #L3 #Cloud #Infrastructure #ALB
🎙️ Candidate Opening & Architectural Context
""AWS reliability requires differentiating between AWS control plane limits and host-level resource exhaustion. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix.""
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: "AWS reliability requires differentiating between AWS control plane limits and host-level resource exhaustion. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix."

1️⃣

Initial Diagnostics & Root Cause Analysis

This sounds like a lifecycle hook or health check issue:

  • Application actually is unhealthy — the new instance really is failing. Check the health check endpoint. Maybe there's a deployment bug.
  • ALB health check failing immediately — instance isn't ready to serve traffic before health checks kick in. The ASG uses ALB health checks and terminates the instance before the app is fully started.
  • Lifecycle hook stuck — a lifecycle hook (autoscaling:EC2_INSTANCE_LAUNCHING) is not completing. Instance is in Pending:Wait state but something is killing it.
2️⃣

Remediation & Permanent Safeguards

Check ASG activity logs in the console — it will say exactly why each instance was terminated.

  • Scale-in protection not set — instance is being terminated by scale-in event despite looking fine.
  • EC2 instance reachability check failing — hardware issue at the hypervisor level. Check AWS console system status checks.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Application actually is unhealthy — the new instance really is failing. Check the health check endpoint. Maybe there's a deploymen."
⚡ 60-Second Elevator Pitch Talking Points
  • Application actually is unhealthy — the new instance really is failing. Check the health check en...
  • ALB health check failing immediately — instance isn't ready to serve traffic before health checks...
  • Lifecycle hook stuck — a lifecycle hook (autoscaling:EC2_INSTANCE_LAUNCHING) is not completing. I...
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Confusing ALB 502 Bad Gateway (target connection reset) with 504 Gateway Timeout (upstream response timeout) when configuring instance Auto Scaling 502 indicates the backend target closed the TCP connection prematurely or returned malformed HTTP, whereas 504 means the target failed to respond before idle timeout.
  2. Setting target deregistration delay to zero instead of allowing connection draining when configuring instance Auto Scaling Instant target deregistration severs in-flight HTTP requests during auto-scaling scale-in or rolling deployments, causing immediate user-facing drops.
  3. Failing to query CloudWatch ELB metrics (HTTPCode_ELB_5XX vs HTTPCode_Target_5XX) during load balancer outages when configuring instance Auto Scaling Comparing load balancer generated errors against target generated errors immediately isolates whether the issue lies in the proxy tier or application runtime.
  4. Overlooking Security Group rules on backend EC2/ECS instances that block health check traffic from the ALB when configuring instance Auto Scaling Target instances must explicitly permit inbound HTTP/HTTPS traffic from the ALB security group on the configured container/application port.
Advertisement
Want more AWS & Cloud Architecture scenarios?
Explore our complete collection of scenario-based AWS & Cloud Architecture interview runbooks.
Browse All AWS & Cloud Architecture Questions →