Q: Pod shows `Terminating` for hours and won't delete.
Force delete: kubectl delete pod <name> --grace-period=0 --force. Usually caused by finalizers or stuck volumes.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: When a <strong>pod shows Terminating for hours</strong> and won't delete, the API object has a deletion timestamp but something is blocking its removal: a finalizer, a kubelet that cannot confirm the container stopped, a volume that will not unmount, or a process ignoring SIGTERM past a long grace period. Diagnose which of these applies before you force anything. Interviewers ask this because the lazy answer, force delete, can leave a StatefulSet pod running twice, and a senior answer explains the safe path.
How Pod Deletion Works and Where It Blocks
Deletion is a sequence; stuck pods are stuck at a specific step.
- Sequence: the API sets
deletionTimestamp, the kubelet runspreStophooks and sends SIGTERM, waitsterminationGracePeriodSeconds(default 30), sends SIGKILL, unmounts volumes, and reports back so the object can be removed. - Finalizers: any string in
metadata.finalizerskeeps the object until its controller removes it. - Node dependency: only the kubelet on the pod's node can confirm termination; if the node is NotReady or unreachable the pod stays Terminating. This is why pods on a failed node show Terminating while a Deployment has already created replacements elsewhere, and why StatefulSet replacements wait.
- Volumes and runtime: a hung CSI unmount or an unresponsive container runtime stalls the kubelet's cleanup.
- Grace period surprises: a pod can legitimately be Terminating for as long as its grace period, so confirm
terminationGracePeriodSecondsis not set to hours before assuming it is stuck.
Hands-on: Pod Stuck in Terminating Diagnosis
Collect four facts: finalizers, node health, grace period and events.
kubectl get pod my-pod -n prod -o jsonpath='{.metadata.deletionTimestamp}{"\n"}{.metadata.deletionGracePeriodSeconds}{"\n"}{.metadata.finalizers}{"\n"}'
kubectl get pod my-pod -n prod -o wide # which node?
kubectl get node <node-name>
kubectl describe pod my-pod -n prod | tail -20
# On the node
sudo journalctl -u kubelet --since "1 hour ago" | grep -i my-pod
sudo crictl ps -a | grep my-pod
- Finalizers and timestamps: if finalizers are present, find the owning controller and why it is not clearing them.
- Node state: NotReady or missing nodes explain most hours-long cases. A node that is NotReady will not respond at all; check it from the cloud console or SSM before assuming Kubernetes is the problem.
- Events and kubelet logs: unmount and kill errors appear on the pod or in the node's kubelet journal.
Root Causes and Targeted Fixes
Fix the cause, not the symptom, where you can.
- Finalizer owner not running: a deleted controller or webhook leaves finalizers behind. Restore the controller, or if it is truly gone, remove only that finalizer. Typical examples are controllers for storage protection or custom resources that were uninstalled before the objects they guarded were cleaned up.
- Node NotReady: recover the node, or drain and delete the Node object after confirming it is really down. For planned shutdowns that were not graceful, Kubernetes 1.28 made non-graceful node shutdown GA: taint the node
node.kubernetes.io/out-of-service=nodeshutdown:NoExecuteto let pods and volume attachments be cleaned up. - App ignores SIGTERM: the process runs as a shell child or never handles the signal, so it waits until SIGKILL after a long grace period. Fix the entrypoint with exec form or an init such as tini.
- Stuck volume unmount: inspect VolumeAttachments with
kubectl get volumeattachmentand CSI driver logs. - Foreground deletion: an owner deleted with foreground propagation waits for dependents, so a Terminating parent object may simply be waiting on children with their own finalizers.
Force Deletion: When and How
Force delete removes the API object without kubelet confirmation, so use it deliberately.
# Last-resort sequence (after confirming the node is down or fenced)
kubectl delete pod my-pod -n prod --grace-period=0 --force
# Orphaned finalizer only
kubectl patch pod my-pod -n prod --type=merge -p '{"metadata":{"finalizers":null}}'
kubectl get pod my-pod -n prod
- Command:
kubectl delete pod my-pod -n prod --grace-period=0 --forceremoves the object immediately; the container may still run on the node. - StatefulSet risk: StatefulSets guarantee at most one pod per identity; force deleting while the old pod may still be running can create two writers to the same volume. Be sure the node is down first.
- Finalizer removal:
kubectl patch pod my-pod -n prod -p '{"metadata":{"finalizers":null}}' --type=mergeis the last resort for orphaned finalizers. - Verify:
kubectl get podreturns NotFound, a replacement is Running and Ready, and no orphaned container remains incrictl pson the old node.
Prevention
Design pods so termination is fast and observable.
- Handle SIGTERM: stop accepting work, drain, exit; keep the grace period just above real drain time.
- Keep preStop short: a sleep or hook that hangs delays termination for the full grace period.
- Monitor: alert on pods with a deletionTimestamp older than a threshold, and use node problem detector for runtime and kubelet failures.
- Control finalizers: uninstall controllers cleanly so they release resources and finalizers first.
- A Terminating pod has a deletion timestamp but is blocked by a finalizer, an unreachable node, a stuck volume or an unhandled SIGTERM.
- I read finalizers, node health, grace period and events before acting.
- I force delete only when the old container is confirmed dead, and I fix SIGTERM handling and monitoring to prevent repeats.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Running force delete as the first step It hides the cause and on StatefulSets can create duplicate writers to one volume.
- Stripping finalizers without finding their owner The owner controller may then leave external resources behind.
- Ignoring node health A NotReady node can never confirm termination, so no pod-level fix will work.
- Using a shell-form entrypoint SIGTERM goes to the shell, not the app, so the app only dies at SIGKILL after the full grace period.