Q: A node in your cluster shows `NotReady`. Your team is panicking because several services are on it. What's your action plan?
Don't panic — check if pods already rescheduled. Kubernetes evicts pods from NotReady nodes after pod-eviction-timeout (default 5 min). Check kubectl get pods -A -o wide | grep .
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "When troubleshooting Kubernetes, I always follow a structured layered model: Pod status -> Events -> Logs -> Network. The interviewer is testing: Incident response, node troubleshooting, pod eviction understanding.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Initial Diagnostics & Root Cause Analysis
- Don't panic — check if pods already rescheduled. Kubernetes evicts pods from NotReady nodes after
pod-eviction-timeout(default 5 min). Checkkubectl get pods -A -o wide | grep. - Cordon the node —
kubectl cordonprevents new pods from scheduling there while you investigate. - SSH into the node and check:
systemctl status kubelet— is kubelet running?journalctl -u kubelet -n 100— kubelet logs.
Remediation & Permanent Safeguards
Execute the resolution runbook and verify workload health:
- Disk space:
df -h. Full disk is a common cause. - Memory:
free -m. - Check the node's conditions:
kubectl describe node— look for MemoryPressure, DiskPressure, PIDPressure. - If unrecoverable, drain and delete:
kubectl drainthen terminate the VM.--ignore-daemonsets --delete-emptydir-data
- Don't panic — check if pods already rescheduled. Kubernetes evicts pods from NotReady nodes after...
- Cordon the node — kubectl cordon prevents new pods from scheduling there while you investigate.
- SSH into the node and check:
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
-
Configuring Service selector labels that mismatch underlying Pod manifest metadata when configuring node cluster shows
A typo in
spec.selectorcauses the Service endpoints list (kubectl get endpoints) to remain empty, resulting in503 Service Unavailableor timeouts. -
Omitting explicit ingressClassName specifications when running multiple ingress controllers when configuring node cluster shows
Without an explicit
ingressClassName, ingress resources are either ignored or picked up unpredictably by competing controllers. -
Depleting worker node subnet IP address allocations under VPC CNI or Azure CNI when configuring node cluster shows
Direct VPC pod networking allocates real subnet IPs; failing to monitor IP pools causes new pods to stay stuck indefinitely in
ContainerCreating. - Attempting network triage without inspecting kubectl describe service and EndpointSlice resources when configuring node cluster shows Verifying EndpointSlice population immediately determines whether a connectivity failure is caused by pod labeling, readiness probes, or routing.