The Production Kubernetes Troubleshooting Handbook
Over 70% of Kubernetes interview questions test your ability to debug failing workloads. This handbook provides the exact mental model, diagnostic matrix, and CLI command sequences used by Lead SREs to triage production clusters.
The Master Kubernetes Diagnostic Matrix
| Pod Status | Root Cause | Diagnostic Command | Resolution Runbook |
|---|---|---|---|
| Node Not Ready | Kubelet crash, containerd CRI deadlock, TLS cert expired, CNI IP exhaustion | journalctl -u kubelet && crictl info | Read Runbook → |
| Node Disk Pressure | imagefs saturation, unpruned images, runaway /var/log/pods, emptyDir leak | df -hT && crictl rmi --prune | Read Runbook → |
| CreateContainerError | CRI sandbox creation failed, EBS Multi-Attach deadlock, subPath missing, AppArmor | kubectl describe pod (Waiting Reason) | Read Runbook → |
| CreateContainerConfigError | Missing ConfigMap or Secret, key absent, namespace mismatch, ESO sync lag | kubectl get configmap,secret -n <ns> | Read Runbook → |
| ImagePullBackOff | ECR auth token expired, Docker Hub 429 rate limit, VPC NAT/S3 endpoint, ARM64 vs AMD64 | kubectl describe pod | grep Events | Read Runbook → |
| RunContainerError | Kernel execve failure: Windows CRLF shebang ( ), missing chmod +x, Distroless shell, postStart hook | kubectl describe pod (failed to start shim) | Read Runbook → |
| OOMKilled (Exit 137) | Container exceeded cgroups memory limit (working_set), JVM container blindness, QoS sacrifice | kubectl describe pod (Last State: Exit 137) | Read Runbook → |
Hands-On Lab & Incident Runbook
Launch Lab Runbook ↗
Top 7 Kubernetes Troubleshooting Use Cases (Lab Runbook)
Step-by-step verified diagnostic CLI sequences, kernel internals, and production YAML manifests on lab.naveedkumbhar.com.
Essential Golden Commands for the Interview
# 1. View previous container crash logs
kubectl logs <pod-name> -c <container> --previous --tail=100
# 2. Inspect real-time scheduler and kubelet events
kubectl get events -n <namespace> --sort-by='.lastTimestamp'
# 3. Fast diagnostic exec with network tools
kubectl debug pod/<pod-name> -it --image=nicolaka/netshoot -- /bin/bash
# 4. Check cluster node resource commitments
kubectl top nodes && kubectl top pods -A --sort-by='memory'