Q: What happens to pods when you run `kubectl drain` on a node?
kubectl drain does two things:
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "When troubleshooting Kubernetes, I always follow a structured layered model: Pod status -> Events -> Logs -> Network. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix."
Initial Diagnostics & Root Cause Analysis
kubectl drain does two things:
- Cordons the node — marks it unschedulable so no new pods land on it.
- Evicts all pods — sends eviction requests (not delete) for every pod. Eviction respects PodDisruptionBudgets.
- DaemonSet pods (need
--ignore-daemonsetsflag to proceed).
Remediation & Permanent Safeguards
Exceptions — these pods are NOT evicted by default: After drain: the node is empty and safe to maintain/terminate.
- Pods with local storage (need
--delete-emptydir-dataflag). - Pods not managed by a controller (standalone pods) — drain will fail unless you force it.
- Cordons the node — marks it unschedulable so no new pods land on it.
- Evicts all pods — sends eviction requests (not delete) for every pod. Eviction respects PodDisrup...
- DaemonSet pods (need --ignore-daemonsets flag to proceed).
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Attempting multi-minor Kubernetes version jumps directly during cluster upgrades when configuring happens pods kubectl Kubernetes explicitly does not support multi-minor version jumps; control plane and worker components require sequential hops to validate API deprecations.
- Upgrading kube-apiserver while leaving critical cluster add-ons and CNI plugins desynchronized when configuring happens pods kubectl Upgrading kube-apiserver without synchronizing CNI daemonsets or CoreDNS manifests causes silent pod networking failure and DNS timeouts immediately post-upgrade.
- Replacing worker node pools simultaneously rather than adopting sequential rolling node group updates when configuring happens pods kubectl In-place or simultaneous node pool replacements risk cluster-wide capacity starvation if new AMIs or kubelet versions encounter bootstrapping errors.
- Overlooking terminationGracePeriodSeconds when evicting stateful database or queue consumer pods when configuring happens pods kubectl Prematurely terminating stateful pods before in-flight transactions commit causes data corruption and unclean lock releases.