Q: In production, how would you move workloads from an unhealthy node to a healthy node, and what is the difference between taint and cordon?
Production runbook for safely evacuating workloads from unhealthy or degrading Kubernetes worker nodes without violating PodDisruptionBudgets or causing service outages.
#Kubernetes #Node Operations #Drain #Cordon #Taint #PDB
🎙️ Candidate Opening & Architectural Context
"In production, evacuating an unhealthy node requires sequential coordination: cordoning prevents new pods from scheduling, draining evicts existing workloads while respecting PodDisruptionBudgets (PDBs), and taints provide granular condition-based repulsion."
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1
Difference: Cordon vs. Taint
kubectl cordon marks a node as unschedulable (node.spec.unschedulable = true), affecting ALL pods uniformly. A Taint (key=value:effect) repels pods unless they have an explicit matching Toleration, allowing fine-grained workload placement.
# Cordon (binary unschedulable)
kubectl cordon ip-10-0-1-50.ec2.internal
# Taint (granular repulsion with Tolerations)
kubectl taint nodes ip-10-0-1-50.ec2.internal maintenance=true:NoSchedule
2
Production Evacuation Workflow (kubectl drain)
Always use kubectl drain for node evacuation. It automatically cordons the node and issues graceful eviction API requests that respect PodDisruptionBudgets, terminating pods gracefully.
kubectl drain ip-10-0-1-50.ec2.internal \
--ignore-daemonsets \
--delete-emptydir-data \
--grace-period=60
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Cordon makes a node unschedulable for all pods; Taints allow exceptions via Tolerations; Drain cordons and safely evicts existing pods obeying PDBs."
⚡ 60-Second Elevator Pitch Talking Points
- kubectl cordon sets unschedulable: true, stopping new pod placement without evicting running pods.
- kubectl taint applies key=value:effect to repel pods unless they hold an explicit matching toleration.
- kubectl drain cordons the node and safely evicts existing pods while respecting PodDisruptionBudgets.
- Always include --ignore-daemonsets and verify cluster capacity before draining.
Advertisement