⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Linux & SRE Interview Questions Scenario 16 of 135 in Linux & SRE
Senior DevOps / SRE [L2] General DevOps General DevOps — Scenario-Based Interview Questions Production Scenario [L2]
🎯 Target Role / Context: General DevOps Scenario #10 (L2) · General DevOps — Scenario-Based Interview Questions

Q: Your Auto Scaling Group (ASG) keeps starting new EC2 instances, but they immediately fail the load balancer health checks and are subsequently terminated by the ASG in an infinite "thrashing" loop. How do you debug this?

If the instance is terminated immediately upon failing the health check, you never get a chance to SSH in and look at the logs.

#General DevOps #L2 #DevOps #SRE #Architecture
🎙️ Candidate Opening & Architectural Context
""When an interviewer asks about this scenario, I explain how we balanced incident response with long-term prevention. The interviewer is testing: ASG lifecycle hooks, scaling failure modes, graceful startup.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third.""
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? The Linux Foundation's Certified System Administrator (LFCS) & SRE Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: "When an interviewer asks about this scenario, I explain how we balanced incident response with long-term prevention. The interviewer is testing: ASG lifecycle hooks, scaling failure modes, graceful startup.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."

1️⃣

Initial Diagnostics & Root Cause Analysis

If the instance is terminated immediately upon failing the health check, you never get a chance to SSH in and look at the logs.

  • Suspend ASG Processes: I would temporarily suspend the Terminate process on the ASG using the AWS Console or CLI. This allows the failing instance to remain alive so I can SSH in, check /var/log/cloud-init-output.log or the application logs, and find the syntax error or missing dependency crashing the boot script.
  • Check Grace Period: The application might naturally take 4 minutes to compile and start, but the ASG Health Check Grace Period might be set to 60 seconds. The ASG kills it prematurely because it hasn't finished booting. I would increase the Grace Period.
  • Lifecycle Hooks: Use an ASG Lifecycle Hook (autoscaling:EC2_INSTANCE_LAUNCHING) to place the instance in a Pending:Wait state while it boots, preventing the ALB from checking it until the instance explicitly signals it is fully ready.
2️⃣

Remediation & Permanent Safeguards

To debug:

💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Suspend ASG Processes: I would temporarily suspend the Terminate process on the ASG using the AWS Console or CLI. This allows the ."
⚡ 60-Second Elevator Pitch Talking Points
  • Suspend ASG Processes: I would temporarily suspend the Terminate process on the ASG using the AWS...
  • Check Grace Period: The application might naturally take 4 minutes to compile and start, but the ...
  • Lifecycle Hooks: Use an ASG Lifecycle Hook (autoscaling:EC2_INSTANCE_LAUNCHING) to place the inst...
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Confusing ALB 502 Bad Gateway (target connection reset) with 504 Gateway Timeout (upstream response timeout) when configuring General Auto Scaling 502 indicates the backend target closed the TCP connection prematurely or returned malformed HTTP, whereas 504 means the target failed to respond before idle timeout.
  2. Setting target deregistration delay to zero instead of allowing connection draining when configuring General Auto Scaling Instant target deregistration severs in-flight HTTP requests during auto-scaling scale-in or rolling deployments, causing immediate user-facing drops.
  3. Failing to query CloudWatch ELB metrics (HTTPCode_ELB_5XX vs HTTPCode_Target_5XX) during load balancer outages when configuring General Auto Scaling Comparing load balancer generated errors against target generated errors immediately isolates whether the issue lies in the proxy tier or application runtime.
  4. Overlooking Security Group rules on backend EC2/ECS instances that block health check traffic from the ALB when configuring General Auto Scaling Target instances must explicitly permit inbound HTTP/HTTPS traffic from the ALB security group on the configured container/application port.
Advertisement
Want more Linux & SRE scenarios?
Explore our complete collection of scenario-based Linux & SRE interview runbooks.
Browse All Linux & SRE Questions →