Q: Pod is stuck in Pending — what would you check?
Step-by-step diagnostic workflow for pods stuck in Pending state: decoding kube-scheduler events, capacity exhaustion, taints/tolerations, PVC binding, and autoscaling response.
Kubernetes Pod Stuck in Pending? Check This First!
Watch Naveed break down how to diagnose and fix a Kubernetes pod stuck in Pending state—decoding kube-scheduler FailedScheduling events, CPU/memory request starvation, taints/tolerations, and unbound PVCs in 60 seconds.
- Run kubectl describe pod and check the Events section for FailedScheduling messages first
- Inspect node allocated resources vs pod CPU/memory requests to isolate capacity exhaustion
- Check node constraints: spec.nodeSelector, nodeAffinity, and taints without tolerations
- Verify persistent storage binding: check PVC status and AWS AZ volume placement constraints
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: A Pod stuck in Pending means the kube-scheduler cannot find a node that meets all the pod's constraints, or volume mounting / admission controllers are blocked. My primary tool is immediately 'kubectl describe pod <pod-name>'.
Inspect Scheduler Events First
Run kubectl describe pod <pod-name> and look directly at the Events section at the bottom:
- FailedScheduling message: The scheduler explains in plain English why each node was rejected (e.g.
0/6 nodes available: 3 Insufficient cpu, 3 node(s) had untolerated taint). - FailedMount / FailedAttachVolume: Storage subsystem is blocked attempting to attach or format an EBS/NFS volume.
Root Cause 1: Insufficient Node Capacity (CPU/Memory)
Scheduler calculates fit based on requests, not actual usage:
- Run
kubectl describe nodes | grep -A 8 'Allocated resources'to see node allocation percentages. - If pods have huge CPU/memory requests (e.g.
cpu: 4on 4-core nodes), no single node can fit them. - Fix: Adjust application requests, or ensure Cluster Autoscaler / Karpenter is provisioning new nodes.
Root Cause 2: Node Selectors, Affinity & Taints
Filter constraints that eliminate eligible nodes:
spec.nodeSelector: Label typo (e.g.disk: ssdwhen nodes are labeleddisktype: ssd).nodeAffinity/podAntiAffinity: Hard anti-affinity (requiredDuringSchedulingIgnoredDuringExecution) preventing pods from running on the same node/AZ.- Taints without Tolerations: Nodes tainted with
dedicated=gpu:NoScheduleor uncordoned maintenance taints.
Root Cause 3: Unbound PVCs & Missing Config
Check persistent storage and required configuration:
- Check PVC state:
kubectl get pvc. IfPending, check StorageClass and CSI provisioner. - Volume Multi-AZ Trap: In AWS, an EBS volume lives in
us-east-1a. If nodes inus-east-1aare full, scheduler cannot place the pod inus-east-1b. - Check referenced ConfigMaps/Secrets: If a volume references a non-existent ConfigMap, pod cannot start.
- Run 'kubectl describe pod <name>' and read the Events section: FailedScheduling tells you why.
- Check Resources: verify node allocated requests ('kubectl describe nodes') vs pod spec.resources.requests.
- Check Constraints: nodeSelector, nodeAffinity, and taints/tolerations that block placement.
- Check Storage: verify PVC status ('kubectl get pvc') - check if volume is locked to a different AWS AZ.
- Verify Cluster Autoscaler / Karpenter logs to ensure nodes are actively spinning up.
- Fix: Tune requests, correct label selectors, add tolerations, or trigger node autoscaling.
Mastered this procedure? Subscribe to @naveedkumbhar01 on YouTube for daily 60-second SRE incident runbooks, Kubernetes triage shortcuts, and high-impact interview elevator pitches.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Restarting a crashing container without examining preceding termination logs via kubectl logs --previous when configuring Kubernetes Stuck Pending Restarting the pod flushes ephemeral container runtime state, destroying vital panic stack traces and fatal initialization logs.
-
Misdiagnosing an application boot crash as a failure of aggressive liveness probes when configuring Kubernetes Stuck Pending
Liveness probes with short timeouts or high failure thresholds kill slow-starting applications repeatedly, causing false
CrashLoopBackOffloops. -
Neglecting node taints, tolerations, and resource quotas when diagnosing pods stuck in Pending when configuring Kubernetes Stuck Pending
Pods remain
Pendingwhen scheduler predicates fail;kubectl describe podreveals whether CPU/memory exhaustion or taints prevent placement. - Invoking kubectl delete pod --force --grace-period=0 indiscriminately on unready pods when configuring Kubernetes Stuck Pending Forced deletion bypasses kubelet cleanup and can cause multi-attach errors on PersistentVolumes (PVs) or split-brain on stateful workloads.