Q: Your EC2 Auto Scaling Group (ASG) dynamically scales down during the night. However, when it terminates an instance, active users downloading large files are abruptly violently disconnected. How do you gracefully drain those connections?
This requires a two-part solution utilizing application load balancing and ASG native hooks:
#AWS #Cost & Architecture #L3 #Cloud #Infrastructure #ALB
🎙️ Candidate Opening & Architectural Context
""AWS reliability requires differentiating between AWS control plane limits and host-level resource exhaustion. The interviewer is testing: ASG Lifecycle Hooks, ALB Deregistration Delay.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third.""
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Initial Diagnostics & Root Cause Analysis
This requires a two-part solution utilizing application load balancing and ASG native hooks:
- ALB Deregistration Delay (Connection Draining): On the ALB Target Group, configure a deregistration delay (e.g., 300 seconds). When the instance is marked for termination, the ALB stops sending *new* requests to it, but keeps the instance alive in a "draining" state allowing active downloads to finish cleanly.
- ASG Lifecycle Hooks: Add a
TerminatingLifecycle Hook to the ASG. This intercepts the EC2 termination command and puts the instance into aTerminating:Waitstate. The instance runs a shutdown script to naturally close stateful background workers, flush caches to Redis, and finally sends aCompleteLifecycleActionAPI call to AWS, allowing the instance to securely power off.
2️⃣
Remediation & Permanent Safeguards
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: ALB Deregistration Delay (Connection Draining): On the ALB Target Group, configure a deregistration delay (e.g., 300 seconds). Whe."
⚡ 60-Second Elevator Pitch Talking Points
- ALB Deregistration Delay (Connection Draining): On the ALB Target Group, configure a deregistrati...
- ASG Lifecycle Hooks: Add a Terminating Lifecycle Hook to the ASG. This intercepts the EC2 termina...
Advertisement