⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 44 of 207 in Kubernetes
Staff SRE / Principal Architect [L3] Kubernetes Storage Staff SRE Scenario [L3]
🎯 Target Role / Context: Kubernetes Scenario #31 (L3) · Storage

Q: A StatefulSet pod can't start because it's trying to attach a volume that's still attached to a terminated pod on a dead node. How do you fix it?

This is a common scenario when a node dies without gracefully releasing its volumes. The PV shows Terminating or the pod shows volume attach error.

#Kubernetes #Storage #L3 #Container Orchestration #K8s #EC2
🎙️ Candidate Opening & Architectural Context
""In one of our high-traffic production clusters, our SRE team handled this incident using a standardized runbook. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix.""
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: "In one of our high-traffic production clusters, our SRE team handled this incident using a standardized runbook. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix."

1️⃣

Initial Diagnostics & Root Cause Analysis

This is a common scenario when a node dies without gracefully releasing its volumes. The PV shows Terminating or the pod shows volume attach error.

  • Force delete the stuck pod: kubectl delete pod --grace-period=0 --force
  • Check if the PV is stuck: kubectl describe pv — look for the node it's attached to.
  • On AWS (EBS): use AWS CLI to force detach the volume: aws ec2 detach-volume --volume-id --force
2️⃣

Remediation & Permanent Safeguards

Steps:

  • Check the VolumeAttachment object: kubectl get volumeattachment — delete the stuck one.
  • The new pod should then attach the volume successfully.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Force delete the stuck pod: kubectl delete pod --grace-period=0 --force."
⚡ 60-Second Elevator Pitch Talking Points
  • Force delete the stuck pod: kubectl delete pod --grace-period=0 --force
  • Check if the PV is stuck: kubectl describe pv — look for the node it's attached to.
  • On AWS (EBS): use AWS CLI to force detach the volume: aws ec2 detach-volume --volume-id --force
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Over-granting cluster-admin permissions to pod ServiceAccounts instead of scoping namespace Roles when configuring StatefulSet start trying Over-granting cluster-wide permissions violates least-privilege principles and allows compromised pods to escalate privileges across namespaces.
  2. Referencing mutable latest container tags across production Kubernetes Deployment specs when configuring StatefulSet start trying Deployments referencing :latest risk inconsistent versions across replica pods and break deterministic rollback capabilities.
  3. Omitting pod readiness probes, causing traffic to hit uninitialized application containers when configuring StatefulSet start trying Readiness probes gate traffic delivery; missing probes cause Kubernetes to route user requests to pods before dependencies or caches are loaded.
  4. Executing diagnostic triage without analyzing chronological cluster events and describe outputs when configuring StatefulSet start trying Cluster events provide sequential timestamps of scheduler decisions, probe failures, and image pull timeouts essential for rapid root-cause isolation.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →