Q: During an EKS deployment across multiple namespaces, one pod gets stuck in Pending state. Walk me through your troubleshooting approach.
Systematic diagnostic workflow when a Kubernetes deployment across multiple namespaces leaves exactly one pod stuck in Pending state.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: When a multi-namespace EKS rollout succeeds across almost all services but leaves a single pod stuck in Pending, the Kubernetes scheduler cannot find a suitable worker node that satisfies that specific pod's resource requests, node selectors, taints/tolerations, or VPC IP allocations.
Execute kubectl describe pod for Scheduler Verdict
Inspect the Events table in kubectl describe pod . The default-scheduler explicitly records why each candidate node was rejected.
kubectl describe pod <pod-name> -n <namespace>
# Common output:
# 0/12 nodes are available: 3 Insufficient cpu, 5 node(s) had untolerated taint, 4 node(s) didn't match PodTopologySpread.
Check Node Affinity, Selectors & Taints/Tolerations
Check whether the failing pod spec has specific constraints that other namespaces don't:
- nodeSelector or nodeAffinity targeting an instance group that doesn't exist (e.g. topology.kubernetes.io/zone=us-east-1a).
- Missing tolerations for dedicated node pool taints (e.g. dedicated=gpu:NoSchedule or spot=true:NoSchedule).
kubectl get nodes --show-labels
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
Check AWS VPC CNI Subnet IP Exhaustion
On Amazon EKS using AWS VPC CNI, every pod receives a real secondary private IP from the node's VPC subnet. If one specific Availability Zone's subnet runs out of available IP addresses (AvailableIpAddressCount: 0), new pods targeting that subnet are stuck in Pending.
aws ec2 describe-subnets \
--subnet-ids subnet-0123456789abcdef0 \
--query "Subnets[*].[SubnetId,AvailableIpAddressCount,CidrBlock]"
Check Resource Requests vs. Cluster Autoscaler / Karpenter
Check if the pod requests excessive CPU or Memory (requests.memory: 32Gi) that exceeds the capacity of available node sizes. Check whether Karpenter or Cluster Autoscaler is failing to spin up new nodes due to AWS EC2 service quotas.
- Run kubectl describe pod to inspect the scheduler's rejection reasons.
- Verify nodeSelectors, nodeAffinity rules, and taints/tolerations on worker node groups.
- Check AWS VPC subnet IP availability to ensure the CNI has free IP addresses for new pods.
- Inspect Karpenter/Cluster Autoscaler logs to verify if autoscaling is blocked by AWS capacity quotas.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Restarting a crashing container without examining preceding termination logs via kubectl logs --previous when addressing multi-namespace Restarting the pod flushes ephemeral container runtime state, destroying vital panic stack traces and fatal initialization logs.
-
Misdiagnosing an application boot crash as a failure of aggressive liveness probes
Liveness probes with short timeouts or high failure thresholds kill slow-starting applications repeatedly, causing false
CrashLoopBackOffloops. -
Neglecting node taints, tolerations, and resource quotas when diagnosing pods stuck in Pending
Pods remain
Pendingwhen scheduler predicates fail;kubectl describe podreveals whether CPU/memory exhaustion or taints prevent placement. - Invoking kubectl delete pod --force --grace-period=0 indiscriminately on unready pods Forced deletion bypasses kubelet cleanup and can cause multi-attach errors on PersistentVolumes (PVs) or split-brain on stateful workloads.