Q: A StatefulSet pod can't start because it's trying to attach a volume that's still attached to a terminated pod on a dead node. How do you fix it?
This is a common scenario when a node dies without gracefully releasing its volumes. The PV shows Terminating or the pod shows volume att...
#Kubernetes #Storage #L3 #Container Orchestration #K8s #EC2
🎙️ Candidate Opening & Architectural Context
""In one of our high-traffic production clusters, our SRE team handled this incident using a standardized runbook. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix.""
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Initial Diagnostics & Root Cause Analysis
This is a common scenario when a node dies without gracefully releasing its volumes. The PV shows Terminating or the pod shows volume attach error.
- Force delete the stuck pod:
kubectl delete pod--grace-period=0 --force - Check if the PV is stuck:
kubectl describe pv— look for the node it's attached to. - On AWS (EBS): use AWS CLI to force detach the volume:
aws ec2 detach-volume --volume-id--force
2️⃣
Remediation & Permanent Safeguards
Steps: --- ## 🟣 RBAC & Security
- Check the VolumeAttachment object:
kubectl get volumeattachment— delete the stuck one. - The new pod should then attach the volume successfully.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Force delete the stuck pod: kubectl delete pod --grace-period=0 --force."
⚡ 60-Second Elevator Pitch Talking Points
- Force delete the stuck pod: kubectl delete pod --grace-period=0 --force
- Check if the PV is stuck: kubectl describe pv — look for the node it's attached to.
- On AWS (EBS): use AWS CLI to force detach the volume: aws ec2 detach-volume --volume-id --force
Advertisement