Q: A node in your cluster shows `NotReady`. Your team is panicking because several services are on it. What's your action plan?
1. Don't panic — check if pods already rescheduled. Kubernetes evicts pods from NotReady nodes after pod-eviction-timeout (default 5 min)...
#Kubernetes #Troubleshooting & Debugging #L3 #Container Orchestration #K8s
🎙️ Candidate Opening & Architectural Context
""When troubleshooting Kubernetes, I always follow a structured layered model: Pod status -> Events -> Logs -> Network. The interviewer is testing: Incident response, node troubleshooting, pod eviction understanding.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third.""
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Initial Diagnostics & Root Cause Analysis
- Don't panic — check if pods already rescheduled. Kubernetes evicts pods from NotReady nodes after
pod-eviction-timeout(default 5 min). Checkkubectl get pods -A -o wide | grep. - Cordon the node —
kubectl cordonprevents new pods from scheduling there while you investigate. - SSH into the node and check:
systemctl status kubelet— is kubelet running?journalctl -u kubelet -n 100— kubelet logs.
2️⃣
Remediation & Permanent Safeguards
Execute the resolution runbook and verify workload health:
- Disk space:
df -h. Full disk is a common cause. - Memory:
free -m. - Check the node's conditions:
kubectl describe node— look for MemoryPressure, DiskPressure, PIDPressure. - If unrecoverable, drain and delete:
kubectl drainthen terminate the VM.--ignore-daemonsets --delete-emptydir-data
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Don't panic — check if pods already rescheduled. Kubernetes evicts pods from NotReady nodes after pod-eviction-timeout (default 5 ."
⚡ 60-Second Elevator Pitch Talking Points
- Don't panic — check if pods already rescheduled. Kubernetes evicts pods from NotReady nodes after...
- Cordon the node — kubectl cordon prevents new pods from scheduling there while you investigate.
- SSH into the node and check:
Advertisement