Q: A Kubernetes pod is stuck in the Pending state due to a storage-related issue. What possible causes would you investigate, and how would you troubleshoot the issue?
Complete troubleshooting workflow when a Kubernetes pod is stuck in Pending state due to PersistentVolumeClaim (PVC) and CSI storage issues.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: When a pod remains in Pending status with storage warnings, the Kubernetes scheduler cannot place the pod onto any node because persistent storage volume constraints cannot be satisfied. I troubleshoot by checking pod events, PVC status, StorageClass provisioners, and Availability Zone topology constraints.
Inspect Pod Describe Events
Run kubectl describe pod . Check the Events section at the bottom. Common storage failure messages:
- 0/3 nodes are available: 3 node(s) had volume node affinity conflict
- pod has unbound immediate PersistentVolumeClaims
- waiting for a volume to be created, either by dynamic provisioning or manual binding
kubectl describe pod <pod-name> -n production
# Look for Events:
# Warning FailedScheduling pod has unbound immediate PersistentVolumeClaims
Investigate PersistentVolumeClaim (PVC) Binding Status
Check whether the PVC is Bound, Pending, or Lost. Describe the PVC to see why dynamic volume provisioning failed.
kubectl get pvc -n production
kubectl describe pvc <pvc-name> -n production
# Typical error:
# StorageClass "gp3" not found OR VolumeBindingFailed: waiting for first consumer to be created
Top Storage Failure Causes
Investigate the top 4 storage failure modes:
1. StorageClass Misconfiguration: StorageClass does not exist or the underlying cloud CSI driver (e.g. ebs.csi.aws.com) is crashed.
2. Availability Zone Affinity Conflict: EBS/Azure Disks are zonal resources. If an EBS volume exists in us-east-1a, but worker nodes with available CPU/RAM are in us-east-1b, the pod cannot schedule.
3. AccessMode Mismatch: Claim requests ReadWriteMany (RWX) on an EBS or Azure Disk volume that only supports ReadWriteOnce (RWO).
4. Volume Limits Exceeded: Worker node has reached maximum EBS volume attachments per EC2 instance (e.g. 28 volumes per nitro instance).
Check CSI Driver DaemonSet Pods
Verify that the cloud storage CSI controller and node daemonset pods are healthy in kube-system.
kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver
- Run kubectl describe pod and kubectl describe pvc to identify the exact storage scheduler error.
- Verify whether the PersistentVolume is locked in a different Availability Zone than available nodes.
- Check AccessModes: EBS volumes support ReadWriteOnce, not ReadWriteMany.
- Verify that the cloud CSI driver pods are healthy and running in kube-system.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
-
Conflating container cgroup OOMKill events (Exit Code 137) with node-level MemoryPressure evictions when addressing storage-related
Exit Code 137 indicates the container cgroup exceeded its
limits.memory, whereas Node MemoryPressure triggers kubelet eviction of lower QoS pods. - Increasing memory limits arbitrarily without profiling application heap allocation and memory leak roots Increasing limits for applications with memory leaks merely delays the inevitable OOMKill and increases host resource waste.
- Overlooking imagePullSecrets and registry credential expiration during ImagePullBackOff triage Missing or expired private registry authentication tokens block container image pulls before pod bootstrapping can begin.
- Omitting startup probes on slow-initializing JVM or inference workloads in Kubernetes Startup probes protect slow-starting applications by pausing liveness and readiness probe evaluation until initial warmup completes.