Q: In production, how would you move workloads from an unhealthy node to a healthy node, and what is the difference between taint and cordon?
Production runbook for safely evacuating workloads from unhealthy or degrading Kubernetes worker nodes without violating PodDisruptionBudgets or causing service outages.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: In production, evacuating an unhealthy node requires sequential coordination: cordoning prevents new pods from scheduling, draining evicts existing workloads while respecting PodDisruptionBudgets (PDBs), and taints provide granular condition-based repulsion.
Difference: Cordon vs. Taint
kubectl cordon marks a node as unschedulable (node.spec.unschedulable = true), affecting ALL pods uniformly. A Taint (key=value:effect) repels pods unless they have an explicit matching Toleration, allowing fine-grained workload placement.
# Cordon (binary unschedulable)
kubectl cordon ip-10-0-1-50.ec2.internal
# Taint (granular repulsion with Tolerations)
kubectl taint nodes ip-10-0-1-50.ec2.internal maintenance=true:NoSchedule
Production Evacuation Workflow (kubectl drain)
Always use kubectl drain for node evacuation. It automatically cordons the node and issues graceful eviction API requests that respect PodDisruptionBudgets, terminating pods gracefully.
kubectl drain ip-10-0-1-50.ec2.internal \
--ignore-daemonsets \
--delete-emptydir-data \
--grace-period=60
- kubectl cordon sets unschedulable: true, stopping new pod placement without evicting running pods.
- kubectl taint applies key=value:effect to repel pods unless they hold an explicit matching toleration.
- kubectl drain cordons the node and safely evicts existing pods while respecting PodDisruptionBudgets.
- Always include --ignore-daemonsets and verify cluster capacity before draining.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Attempting multi-minor Kubernetes version jumps directly during cluster upgrades Kubernetes explicitly does not support multi-minor version jumps; control plane and worker components require sequential hops to validate API deprecations.
- Upgrading kube-apiserver while leaving critical cluster add-ons and CNI plugins desynchronized Upgrading kube-apiserver without synchronizing CNI daemonsets or CoreDNS manifests causes silent pod networking failure and DNS timeouts immediately post-upgrade.
- Replacing worker node pools simultaneously rather than adopting sequential rolling node group updates In-place or simultaneous node pool replacements risk cluster-wide capacity starvation if new AMIs or kubelet versions encounter bootstrapping errors.
- Overlooking terminationGracePeriodSeconds when evicting stateful database or queue consumer pods Prematurely terminating stateful pods before in-flight transactions commit causes data corruption and unclean lock releases.