⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 28 of 207 in Kubernetes
Senior DevOps / SRE [L2] Kubernetes Deployments & Workloads Production Scenario [L2]
🎯 Target Role / Context: Kubernetes Scenario #15 (L2) · Deployments & Workloads

Q: You want to make sure two pods of the same app NEVER run on the same node. How?

Use Pod Anti-Affinity:

#Kubernetes #Deployments & Workloads #L2 #Container Orchestration #K8s
🎙️ Candidate Opening & Architectural Context
"To make sure two pods of the same app <strong>never run on the same node</strong>, add a <code>requiredDuringSchedulingIgnoredDuringExecution</code> pod anti-affinity rule that matches the app's own labels and uses <code>kubernetes.io/hostname</code> as the <code>topologyKey</code>. The scheduler will then refuse to place a second matching pod on any node that already runs one. Interviewers ask this to see whether you know the difference between a hard guarantee and a preference, and whether you have thought about what happens when replicas outnumber nodes or when a rolling update needs surge capacity."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: To make sure two pods of the same app <strong>never run on the same node</strong>, add a <code>requiredDuringSchedulingIgnoredDuringExecution</code> pod anti-affinity rule that matches the app's own labels and uses <code>kubernetes.io/hostname</code> as the <code>topologyKey</code>. The scheduler will then refuse to place a second matching pod on any node that already runs one. Interviewers ask this to see whether you know the difference between a hard guarantee and a preference, and whether you have thought about what happens when replicas outnumber nodes or when a rolling update needs surge capacity.

1️⃣

Mental Model: Hard Anti-Affinity vs Soft Preference

Pod anti-affinity is evaluated by the scheduler at placement time, against pods that already exist in a topology domain.

  • Required (hard): requiredDuringSchedulingIgnoredDuringExecution is a filter. If no node satisfies it, the pod stays Pending. This is the only form that gives a true guarantee.
  • Preferred (soft): preferredDuringSchedulingIgnoredDuringExecution only adds a score weight (1 to 100). The scheduler may still co-locate pods when nothing else fits, so it does not satisfy a NEVER requirement.
  • topologyKey: defines what counts as the same place. kubernetes.io/hostname means one node; topology.kubernetes.io/zone means one availability zone.
  • IgnoredDuringExecution: the rule is not re-checked after scheduling. If someone relabels a pod or a node, running pods are not evicted.
  • Why hostname: every node carries the well-known label kubernetes.io/hostname with a unique value, so using it as the topologyKey makes each node its own domain.
Pro Tip: A hard rule is a trade of availability for isolation: if you run 4 replicas on a 3 node cluster, the fourth pod will sit in <code>Pending</code> forever. That is the correct behaviour for a NEVER requirement, so size the cluster for it.
2️⃣

Hands-on: Pods Never Run on the Same Node (YAML)

The labelSelector must match the labels of the pods being created, otherwise the rule constrains nothing.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: prod
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app: my-app
              topologyKey: kubernetes.io/hostname
      containers:
        - name: my-app
          image: registry.example.com/my-app:1.4.2
          resources:
            requests: { cpu: 100m, memory: 128Mi }
  • Selector match: the labelSelector below matches app: my-app, which is also in the pod template labels. A typo here is the most common reason the rule silently does nothing.
  • Scope: without namespaces or namespaceSelector in the term, only pods in the pod's own namespace are considered.
  • Apply and watch: after applying, kubectl get pods -l app=my-app -o wide should show a different value in the NODE column for every replica.
Advertisement
3️⃣

Verify Placement and Read Scheduler Events

Never trust the manifest alone; confirm what the scheduler actually did.

kubectl get pods -n prod -l app=my-app \
  -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName,STATUS:.status.phase

# Count replicas per node, anything above 1 is a violation
kubectl get pods -n prod -l app=my-app -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' \
  | sort | uniq -c
  • Check node spread: kubectl get pods -n prod -l app=my-app -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName lists each pod with its node. Any duplicate node name means the rule is not in effect.
  • Read the Pending reason: kubectl describe pod <pod> shows an event like 0/3 nodes are available: 3 node(s) didn't match pod anti-affinity rules. That message confirms the rule is working and capacity is the issue.
  • Confirm the rendered spec: kubectl get deploy my-app -n prod -o jsonpath='{.spec.template.spec.affinity}' proves what was actually applied, which matters when Helm or Kustomize patches are involved.
4️⃣

Failure Modes: Rolling Updates, Replica Count and Autoscaling

Hard anti-affinity interacts badly with default rollout settings and with small clusters.

  • Rolling update deadlock: during a rollout, old and new ReplicaSet pods share the app label, so a surge pod repels itself from every node that holds an old pod. With replicas equal to node count and the default maxSurge: 25%, the surge pod can stay Pending. Set maxSurge: 0 and maxUnavailable: 1, or keep at least one spare node.
  • Replicas greater than nodes: extra pods stay Pending. With Cluster Autoscaler or Karpenter, a new node will be provisioned if one is allowed to satisfy the rule; check the autoscaler logs when pods are Pending but the node group has headroom.
  • Taints and selectors: if only some nodes are eligible (nodeSelector, taints, GPU pools), the effective limit is the eligible node count, not the cluster node count.
  • Alternative: topologySpreadConstraints with maxSkew: 1 and whenUnsatisfiable: DoNotSchedule keeps pods evenly spread and scales better, but it limits imbalance rather than forbidding sharing, so it is not a strict NEVER.
  • Node drains: with hard anti-affinity and no spare node, a drained pod cannot be rescheduled elsewhere and stays Pending until the node returns or capacity is added, so plan maintenance windows with one node of headroom.
5️⃣

Production Practice: Spread, Budgets and Cost of Scheduling

Anti-affinity is one control; availability needs a few more around it.

  • Pair with a PodDisruptionBudget: spreading pods does not stop a node drain from evicting two of them in sequence. A PDB with minAvailable protects voluntary disruptions.
  • Layer zone spread: use required hostname anti-affinity plus a soft or topology-spread rule on topology.kubernetes.io/zone so a single zone failure does not take out the service.
  • Scheduler cost: the Kubernetes docs warn that inter-pod affinity and anti-affinity need significant processing and can slow scheduling in large clusters. Prefer topology spread constraints for big, busy clusters.
  • Enforce by policy: use Kyverno or OPA Gatekeeper to require the rule on tier-1 workloads rather than relying on each team to copy the YAML.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Use required podAntiAffinity on kubernetes.io/hostname for the hard NEVER guarantee, then set maxSurge: 0 on the rollout strategy so surge pods are not blocked by the very rule you wrote."
⚡ 60-Second Elevator Pitch Talking Points
  • Add required podAntiAffinity with topologyKey kubernetes.io/hostname and a labelSelector that matches the app's own pod labels.
  • I verify with kubectl get pods -o wide and by reading the Pending event when replicas exceed nodes.
  • I protect rollouts with maxSurge 0, add a PDB, and enforce the pattern with a Kyverno policy.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Using preferredDuringScheduling for a NEVER requirement Preferred rules are only a scoring hint, so the scheduler will still co-locate pods when nodes are tight.
  2. Writing a labelSelector that does not match the pod template labels The rule then selects no pods and silently enforces nothing, and nobody notices until a node failure.
  3. Running as many replicas as nodes with default maxSurge The surge pod repels itself from every node holding an old pod, so the rollout stalls with a Pending pod.
  4. Forgetting that the rule is not re-evaluated after scheduling IgnoredDuringExecution means existing co-located pods are not moved if labels change later.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →