⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 174 of 207 in Kubernetes
Senior DevOps / SRE Kubernetes Node Operations & Maintenance Production Runbook
🎯 Target Role / Context: Kubernetes Node Operations & Incident Triage

Q: In production, how would you move workloads from an unhealthy node to a healthy node, and what is the difference between taint and cordon?

Production runbook for safely evacuating workloads from unhealthy or degrading Kubernetes worker nodes without violating PodDisruptionBudgets or causing service outages.

#Kubernetes #Node Operations #Drain #Cordon #Taint #PDB
🎙️ Candidate Opening & Architectural Context
"In production, evacuating an unhealthy node requires sequential coordination: cordoning prevents new pods from scheduling, draining evicts existing workloads while respecting PodDisruptionBudgets (PDBs), and taints provide granular condition-based repulsion."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: In production, evacuating an unhealthy node requires sequential coordination: cordoning prevents new pods from scheduling, draining evicts existing workloads while respecting PodDisruptionBudgets (PDBs), and taints provide granular condition-based repulsion.

1

Difference: Cordon vs. Taint

kubectl cordon marks a node as unschedulable (node.spec.unschedulable = true), affecting ALL pods uniformly. A Taint (key=value:effect) repels pods unless they have an explicit matching Toleration, allowing fine-grained workload placement.

# Cordon (binary unschedulable)
kubectl cordon ip-10-0-1-50.ec2.internal

# Taint (granular repulsion with Tolerations)
kubectl taint nodes ip-10-0-1-50.ec2.internal maintenance=true:NoSchedule
2

Production Evacuation Workflow (kubectl drain)

Always use kubectl drain for node evacuation. It automatically cordons the node and issues graceful eviction API requests that respect PodDisruptionBudgets, terminating pods gracefully.

kubectl drain ip-10-0-1-50.ec2.internal \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=60
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Cordon makes a node unschedulable for all pods; Taints allow exceptions via Tolerations; Drain cordons and safely evicts existing pods obeying PDBs."
⚡ 60-Second Elevator Pitch Talking Points
  • kubectl cordon sets unschedulable: true, stopping new pod placement without evicting running pods.
  • kubectl taint applies key=value:effect to repel pods unless they hold an explicit matching toleration.
  • kubectl drain cordons the node and safely evicts existing pods while respecting PodDisruptionBudgets.
  • Always include --ignore-daemonsets and verify cluster capacity before draining.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Attempting multi-minor Kubernetes version jumps directly during cluster upgrades Kubernetes explicitly does not support multi-minor version jumps; control plane and worker components require sequential hops to validate API deprecations.
  2. Upgrading kube-apiserver while leaving critical cluster add-ons and CNI plugins desynchronized Upgrading kube-apiserver without synchronizing CNI daemonsets or CoreDNS manifests causes silent pod networking failure and DNS timeouts immediately post-upgrade.
  3. Replacing worker node pools simultaneously rather than adopting sequential rolling node group updates In-place or simultaneous node pool replacements risk cluster-wide capacity starvation if new AMIs or kubelet versions encounter bootstrapping errors.
  4. Overlooking terminationGracePeriodSeconds when evicting stateful database or queue consumer pods Prematurely terminating stateful pods before in-flight transactions commit causes data corruption and unclean lock releases.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →