⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 92 of 207 in Kubernetes
Senior DevOps / SRE [L2] Kubernetes Advanced Scenarios Production Scenario [L2]
🎯 Target Role / Context: Kubernetes Scenario #82 (L2) · Advanced Scenarios

Q: Pod shows `Terminating` for hours and won't delete.

Force delete: kubectl delete pod <name> --grace-period=0 --force. Usually caused by finalizers or stuck volumes.

#Kubernetes #Advanced Scenarios #L2 #Container Orchestration #K8s
🎙️ Candidate Opening & Architectural Context
"When a <strong>pod shows Terminating for hours</strong> and won't delete, the API object has a deletion timestamp but something is blocking its removal: a finalizer, a kubelet that cannot confirm the container stopped, a volume that will not unmount, or a process ignoring SIGTERM past a long grace period. Diagnose which of these applies before you force anything. Interviewers ask this because the lazy answer, force delete, can leave a StatefulSet pod running twice, and a senior answer explains the safe path."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: When a <strong>pod shows Terminating for hours</strong> and won't delete, the API object has a deletion timestamp but something is blocking its removal: a finalizer, a kubelet that cannot confirm the container stopped, a volume that will not unmount, or a process ignoring SIGTERM past a long grace period. Diagnose which of these applies before you force anything. Interviewers ask this because the lazy answer, force delete, can leave a StatefulSet pod running twice, and a senior answer explains the safe path.

1️⃣

How Pod Deletion Works and Where It Blocks

Deletion is a sequence; stuck pods are stuck at a specific step.

  • Sequence: the API sets deletionTimestamp, the kubelet runs preStop hooks and sends SIGTERM, waits terminationGracePeriodSeconds (default 30), sends SIGKILL, unmounts volumes, and reports back so the object can be removed.
  • Finalizers: any string in metadata.finalizers keeps the object until its controller removes it.
  • Node dependency: only the kubelet on the pod's node can confirm termination; if the node is NotReady or unreachable the pod stays Terminating. This is why pods on a failed node show Terminating while a Deployment has already created replacements elsewhere, and why StatefulSet replacements wait.
  • Volumes and runtime: a hung CSI unmount or an unresponsive container runtime stalls the kubelet's cleanup.
  • Grace period surprises: a pod can legitimately be Terminating for as long as its grace period, so confirm terminationGracePeriodSeconds is not set to hours before assuming it is stuck.
2️⃣

Hands-on: Pod Stuck in Terminating Diagnosis

Collect four facts: finalizers, node health, grace period and events.

kubectl get pod my-pod -n prod -o jsonpath='{.metadata.deletionTimestamp}{"\n"}{.metadata.deletionGracePeriodSeconds}{"\n"}{.metadata.finalizers}{"\n"}'
kubectl get pod my-pod -n prod -o wide      # which node?
kubectl get node <node-name>
kubectl describe pod my-pod -n prod | tail -20

# On the node
sudo journalctl -u kubelet --since "1 hour ago" | grep -i my-pod
sudo crictl ps -a | grep my-pod
  • Finalizers and timestamps: if finalizers are present, find the owning controller and why it is not clearing them.
  • Node state: NotReady or missing nodes explain most hours-long cases. A node that is NotReady will not respond at all; check it from the cloud console or SSM before assuming Kubernetes is the problem.
  • Events and kubelet logs: unmount and kill errors appear on the pod or in the node's kubelet journal.
Advertisement
3️⃣

Root Causes and Targeted Fixes

Fix the cause, not the symptom, where you can.

  • Finalizer owner not running: a deleted controller or webhook leaves finalizers behind. Restore the controller, or if it is truly gone, remove only that finalizer. Typical examples are controllers for storage protection or custom resources that were uninstalled before the objects they guarded were cleaned up.
  • Node NotReady: recover the node, or drain and delete the Node object after confirming it is really down. For planned shutdowns that were not graceful, Kubernetes 1.28 made non-graceful node shutdown GA: taint the node node.kubernetes.io/out-of-service=nodeshutdown:NoExecute to let pods and volume attachments be cleaned up.
  • App ignores SIGTERM: the process runs as a shell child or never handles the signal, so it waits until SIGKILL after a long grace period. Fix the entrypoint with exec form or an init such as tini.
  • Stuck volume unmount: inspect VolumeAttachments with kubectl get volumeattachment and CSI driver logs.
  • Foreground deletion: an owner deleted with foreground propagation waits for dependents, so a Terminating parent object may simply be waiting on children with their own finalizers.
4️⃣

Force Deletion: When and How

Force delete removes the API object without kubelet confirmation, so use it deliberately.

# Last-resort sequence (after confirming the node is down or fenced)
kubectl delete pod my-pod -n prod --grace-period=0 --force
# Orphaned finalizer only
kubectl patch pod my-pod -n prod --type=merge -p '{"metadata":{"finalizers":null}}'
kubectl get pod my-pod -n prod
  • Command: kubectl delete pod my-pod -n prod --grace-period=0 --force removes the object immediately; the container may still run on the node.
  • StatefulSet risk: StatefulSets guarantee at most one pod per identity; force deleting while the old pod may still be running can create two writers to the same volume. Be sure the node is down first.
  • Finalizer removal: kubectl patch pod my-pod -n prod -p '{"metadata":{"finalizers":null}}' --type=merge is the last resort for orphaned finalizers.
  • Verify: kubectl get pod returns NotFound, a replacement is Running and Ready, and no orphaned container remains in crictl ps on the old node.
Pro Tip: Force delete only after you know the old container is dead (node powered off or fenced). For stateful workloads, an unconfirmed force delete risks data corruption.
5️⃣

Prevention

Design pods so termination is fast and observable.

  • Handle SIGTERM: stop accepting work, drain, exit; keep the grace period just above real drain time.
  • Keep preStop short: a sleep or hook that hangs delays termination for the full grace period.
  • Monitor: alert on pods with a deletionTimestamp older than a threshold, and use node problem detector for runtime and kubelet failures.
  • Control finalizers: uninstall controllers cleanly so they release resources and finalizers first.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Check metadata.finalizers and the node status before forcing, because those two facts decide whether a force delete is a safe shortcut or a StatefulSet data hazard."
⚡ 60-Second Elevator Pitch Talking Points
  • A Terminating pod has a deletion timestamp but is blocked by a finalizer, an unreachable node, a stuck volume or an unhandled SIGTERM.
  • I read finalizers, node health, grace period and events before acting.
  • I force delete only when the old container is confirmed dead, and I fix SIGTERM handling and monitoring to prevent repeats.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Running force delete as the first step It hides the cause and on StatefulSets can create duplicate writers to one volume.
  2. Stripping finalizers without finding their owner The owner controller may then leave external resources behind.
  3. Ignoring node health A NotReady node can never confirm termination, so no pod-level fix will work.
  4. Using a shell-form entrypoint SIGTERM goes to the shell, not the app, so the app only dies at SIGKILL after the full grace period.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →