Q: You want to make sure two pods of the same app NEVER run on the same node. How?
Use Pod Anti-Affinity:
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: To make sure two pods of the same app <strong>never run on the same node</strong>, add a <code>requiredDuringSchedulingIgnoredDuringExecution</code> pod anti-affinity rule that matches the app's own labels and uses <code>kubernetes.io/hostname</code> as the <code>topologyKey</code>. The scheduler will then refuse to place a second matching pod on any node that already runs one. Interviewers ask this to see whether you know the difference between a hard guarantee and a preference, and whether you have thought about what happens when replicas outnumber nodes or when a rolling update needs surge capacity.
Mental Model: Hard Anti-Affinity vs Soft Preference
Pod anti-affinity is evaluated by the scheduler at placement time, against pods that already exist in a topology domain.
- Required (hard):
requiredDuringSchedulingIgnoredDuringExecutionis a filter. If no node satisfies it, the pod staysPending. This is the only form that gives a true guarantee. - Preferred (soft):
preferredDuringSchedulingIgnoredDuringExecutiononly adds a score weight (1 to 100). The scheduler may still co-locate pods when nothing else fits, so it does not satisfy a NEVER requirement. - topologyKey: defines what counts as the same place.
kubernetes.io/hostnamemeans one node;topology.kubernetes.io/zonemeans one availability zone. - IgnoredDuringExecution: the rule is not re-checked after scheduling. If someone relabels a pod or a node, running pods are not evicted.
- Why hostname: every node carries the well-known label
kubernetes.io/hostnamewith a unique value, so using it as the topologyKey makes each node its own domain.
Hands-on: Pods Never Run on the Same Node (YAML)
The labelSelector must match the labels of the pods being created, otherwise the rule constrains nothing.
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
namespace: prod
spec:
replicas: 3
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: my-app
topologyKey: kubernetes.io/hostname
containers:
- name: my-app
image: registry.example.com/my-app:1.4.2
resources:
requests: { cpu: 100m, memory: 128Mi }
- Selector match: the
labelSelectorbelow matchesapp: my-app, which is also in the pod template labels. A typo here is the most common reason the rule silently does nothing. - Scope: without
namespacesornamespaceSelectorin the term, only pods in the pod's own namespace are considered. - Apply and watch: after applying,
kubectl get pods -l app=my-app -o wideshould show a different value in theNODEcolumn for every replica.
Verify Placement and Read Scheduler Events
Never trust the manifest alone; confirm what the scheduler actually did.
kubectl get pods -n prod -l app=my-app \
-o custom-columns=POD:.metadata.name,NODE:.spec.nodeName,STATUS:.status.phase
# Count replicas per node, anything above 1 is a violation
kubectl get pods -n prod -l app=my-app -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' \
| sort | uniq -c
- Check node spread:
kubectl get pods -n prod -l app=my-app -o custom-columns=POD:.metadata.name,NODE:.spec.nodeNamelists each pod with its node. Any duplicate node name means the rule is not in effect. - Read the Pending reason:
kubectl describe pod <pod>shows an event like0/3 nodes are available: 3 node(s) didn't match pod anti-affinity rules. That message confirms the rule is working and capacity is the issue. - Confirm the rendered spec:
kubectl get deploy my-app -n prod -o jsonpath='{.spec.template.spec.affinity}'proves what was actually applied, which matters when Helm or Kustomize patches are involved.
Failure Modes: Rolling Updates, Replica Count and Autoscaling
Hard anti-affinity interacts badly with default rollout settings and with small clusters.
- Rolling update deadlock: during a rollout, old and new ReplicaSet pods share the
applabel, so a surge pod repels itself from every node that holds an old pod. With replicas equal to node count and the defaultmaxSurge: 25%, the surge pod can stay Pending. SetmaxSurge: 0andmaxUnavailable: 1, or keep at least one spare node. - Replicas greater than nodes: extra pods stay Pending. With Cluster Autoscaler or Karpenter, a new node will be provisioned if one is allowed to satisfy the rule; check the autoscaler logs when pods are Pending but the node group has headroom.
- Taints and selectors: if only some nodes are eligible (nodeSelector, taints, GPU pools), the effective limit is the eligible node count, not the cluster node count.
- Alternative:
topologySpreadConstraintswithmaxSkew: 1andwhenUnsatisfiable: DoNotSchedulekeeps pods evenly spread and scales better, but it limits imbalance rather than forbidding sharing, so it is not a strict NEVER. - Node drains: with hard anti-affinity and no spare node, a drained pod cannot be rescheduled elsewhere and stays Pending until the node returns or capacity is added, so plan maintenance windows with one node of headroom.
Production Practice: Spread, Budgets and Cost of Scheduling
Anti-affinity is one control; availability needs a few more around it.
- Pair with a PodDisruptionBudget: spreading pods does not stop a node drain from evicting two of them in sequence. A PDB with
minAvailableprotects voluntary disruptions. - Layer zone spread: use required hostname anti-affinity plus a soft or topology-spread rule on
topology.kubernetes.io/zoneso a single zone failure does not take out the service. - Scheduler cost: the Kubernetes docs warn that inter-pod affinity and anti-affinity need significant processing and can slow scheduling in large clusters. Prefer topology spread constraints for big, busy clusters.
- Enforce by policy: use Kyverno or OPA Gatekeeper to require the rule on tier-1 workloads rather than relying on each team to copy the YAML.
- Add required podAntiAffinity with topologyKey kubernetes.io/hostname and a labelSelector that matches the app's own pod labels.
- I verify with kubectl get pods -o wide and by reading the Pending event when replicas exceed nodes.
- I protect rollouts with maxSurge 0, add a PDB, and enforce the pattern with a Kyverno policy.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Using preferredDuringScheduling for a NEVER requirement Preferred rules are only a scoring hint, so the scheduler will still co-locate pods when nodes are tight.
- Writing a labelSelector that does not match the pod template labels The rule then selects no pods and silently enforces nothing, and nobody notices until a node failure.
- Running as many replicas as nodes with default maxSurge The surge pod repels itself from every node holding an old pod, so the rollout stalls with a Pending pod.
- Forgetting that the rule is not re-evaluated after scheduling IgnoredDuringExecution means existing co-located pods are not moved if labels change later.