Q: Your EC2 Auto Scaling Group (ASG) dynamically scales down during the night. However, when it terminates an instance, active users downloading large files are abruptly disconnected. How do you gracefully drain those connections?
This requires a two-part solution utilizing application load balancing and ASG native hooks:
Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "AWS reliability requires differentiating between AWS control plane limits and host-level resource exhaustion. The interviewer is testing: ASG Lifecycle Hooks, ALB Deregistration Delay.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Initial Diagnostics & Root Cause Analysis
This requires a two-part solution utilizing application load balancing and ASG native hooks:
- ALB Deregistration Delay (Connection Draining): On the ALB Target Group, configure a deregistration delay (e.g., 300 seconds). When the instance is marked for termination, the ALB stops sending *new* requests to it, but keeps the instance alive in a "draining" state allowing active downloads to finish cleanly.
- ASG Lifecycle Hooks: Add a
TerminatingLifecycle Hook to the ASG. This intercepts the EC2 termination command and puts the instance into aTerminating:Waitstate. The instance runs a shutdown script to naturally close stateful background workers, flush caches to Redis, and finally sends aCompleteLifecycleActionAPI call to AWS, allowing the instance to securely power off.
Remediation & Permanent Safeguards
- ALB Deregistration Delay (Connection Draining): On the ALB Target Group, configure a deregistrati...
- ASG Lifecycle Hooks: Add a Terminating Lifecycle Hook to the ASG. This intercepts the EC2 termina...
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Confusing ALB 502 Bad Gateway (target connection reset) with 504 Gateway Timeout (upstream response timeout) when configuring Auto Scaling Group 502 indicates the backend target closed the TCP connection prematurely or returned malformed HTTP, whereas 504 means the target failed to respond before idle timeout.
- Setting target deregistration delay to zero instead of allowing connection draining when configuring Auto Scaling Group Instant target deregistration severs in-flight HTTP requests during auto-scaling scale-in or rolling deployments, causing immediate user-facing drops.
-
Failing to query CloudWatch ELB metrics (
HTTPCode_ELB_5XXvsHTTPCode_Target_5XX) during load balancer outages when configuring Auto Scaling Group Comparing load balancer generated errors against target generated errors immediately isolates whether the issue lies in the proxy tier or application runtime. - Overlooking Security Group rules on backend EC2/ECS instances that block health check traffic from the ALB when configuring Auto Scaling Group Target instances must explicitly permit inbound HTTP/HTTPS traffic from the ALB security group on the configured container/application port.