⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 86 of 207 in Kubernetes
Senior DevOps / SRE [L2] Kubernetes Advanced Scenarios Production Scenario [L2]
🎯 Target Role / Context: Kubernetes Scenario #75 (L2) · Advanced Scenarios

Q: Two pods can't communicate even though NetworkPolicy allows it.

CNI plugin may not support NetworkPolicy. Check CNI (Flannel doesn't, Calico does).

#Kubernetes #Advanced Scenarios #L2 #Container Orchestration #K8s
🎙️ Candidate Opening & Architectural Context
"When <strong>two pods can't communicate even though NetworkPolicy allows it</strong>, the rule you are staring at is rarely the problem. Check that your CNI enforces policies at all, that the source pod has a matching egress rule if egress is restricted, that DNS is allowed, that selectors mean AND not OR, and that the port is the pod port, not the Service port. Interviewers ask this because NetworkPolicy semantics are additive, directional and easy to misread, and debugging needs a methodical approach."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: When <strong>two pods can't communicate even though NetworkPolicy allows it</strong>, the rule you are staring at is rarely the problem. Check that your CNI enforces policies at all, that the source pod has a matching egress rule if egress is restricted, that DNS is allowed, that selectors mean AND not OR, and that the port is the pod port, not the Service port. Interviewers ask this because NetworkPolicy semantics are additive, directional and easy to misread, and debugging needs a methodical approach.

1️⃣

Mental Model: How NetworkPolicy Really Decides

A connection needs permission on both ends, and policies only restrict pods they select.

  • Selection: a pod is isolated for ingress or egress only if some policy selects it for that direction. Otherwise it allows everything.
  • Both directions: if the source pod is egress-restricted, you need an egress rule on the source and an ingress rule on the destination. Allowing only one side blocks traffic.
  • Additive: policies are unions of allows; there are no deny rules in the core API, so a stray default-deny policy in the namespace can be the cause.
  • Enforcement: the API server accepts NetworkPolicy objects regardless; only a CNI that implements them (Calico, Cilium, Antrea and others) enforces them. Plain Flannel does not.
  • Stateful tracking: established return traffic is allowed automatically by the CNI, so you only describe who may initiate the connection, in each direction where isolation applies.
2️⃣

Hands-on: Pods Can't Communicate Even Though NetworkPolicy Allows It

Reproduce the failure from the client pod, then list every policy that selects either pod.

kubectl get netpol -A
kubectl describe netpol -n prod
kubectl get pods -n prod --show-labels
kubectl get ns --show-labels | grep -E 'prod|staging'

# Test from the client pod's network namespace
kubectl debug -it -n prod pod/client --image=nicolaka/netshoot --target=client -- \
  sh -c 'nslookup backend.prod.svc.cluster.local; nc -zv backend.prod.svc.cluster.local 8080'
  • List policies: kubectl get netpol -A and kubectl describe netpol -n prod show which pods each policy selects and what it allows.
  • Check labels: kubectl get pods -n prod --show-labels and kubectl get ns --show-labels; policies match labels, and a typo makes a rule select nothing.
  • Test with the right tool: an ephemeral debug container avoids changing the app image.
Advertisement
3️⃣

The Usual Culprits: Why Allowed Traffic Still Fails

These account for most real cases.

# Correct AND semantics: pods labelled app=client in namespaces labelled team=payments
ingress:
  - from:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: payments
        podSelector:
          matchLabels:
            app: client
    ports:
      - protocol: TCP
        port: 8080
  • Missing egress or DNS: a default-deny egress policy blocks DNS to CoreDNS in kube-system (UDP and TCP 53), so names fail even if the target IP is allowed.
  • AND versus OR in from/to: a single list item containing both namespaceSelector and podSelector means AND; two separate dash items mean OR. Mixing them up opens too much or too little.
  • Namespace labels: a namespaceSelector needs a label on the namespace itself. Kubernetes sets kubernetes.io/metadata.name automatically, which is the reliable one to use.
  • Wrong port: policies match the destination pod's container port, not the Service port. If the Service maps 80 to 8080, allow 8080.
  • Not the CNI you think: check which plugin is installed with kubectl get pods -n kube-system; a non-enforcing plugin means policies are ignored and the real block is elsewhere (Service selector, readiness, mesh).
4️⃣

Verification with CNI Flow Tools

Ask the dataplane what it dropped instead of reasoning from YAML alone.

  • Cilium: hubble observe --namespace prod --verdict DROPPED shows dropped flows and the policy reason; cilium monitor --type drop works on the node.
  • Calico: enable policy logging with a Log action in a staged policy, or inspect with calicoctl and the node's iptables counters.
  • Control test: in a non-production namespace, remove one policy at a time to find the culprit, then restore it.
  • Pod IP versus Service: test the pod IP directly; if that works and the Service name fails, the issue is the Service, DNS or kube-proxy, not NetworkPolicy.
  • Host and probe traffic: kubelet health probes and hostNetwork pods are treated specially by many CNIs, so a policy that looks correct can still behave differently for node-originated traffic.
5️⃣

Production Practice

Make policy changes testable and reversible.

  • Default deny plus explicit allows: include DNS egress in the baseline policy so teams do not break name resolution.
  • Test in CI: tools such as cyclonus or Cilium connectivity tests validate that policies behave as intended in a kind cluster.
  • Observability: keep flow logs (Hubble or equivalent) enabled so a dropped connection is a query, not a guess.
  • Review checklist: for every new policy ask which pods it selects, which direction it covers, whether DNS is allowed, and what a test connection from an unauthorised pod should show.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Look for a default-deny egress policy on the source pod and for missing DNS egress first; allowing ingress on the destination alone never fixes a connection that dies at the client's egress."
⚡ 60-Second Elevator Pitch Talking Points
  • NetworkPolicy is directional and additive, and only a CNI that implements it enforces it.
  • I list every policy selecting either pod, check egress and DNS, verify label and port matches, then test with a debug container.
  • I confirm drops with Hubble or Calico logs and bake a DNS-allowing default-deny baseline into CI-tested policies.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Only writing an ingress rule on the destination If the source is egress-restricted, its own egress policy still blocks the connection.
  2. Putting namespaceSelector and podSelector in separate list items That is OR, not AND, so the rule allows far more than intended or something else than expected.
  3. Using the Service port in the policy Policies are evaluated against the pod's target port after Service translation.
  4. Assuming the cluster enforces NetworkPolicy On a CNI without policy support the objects are accepted but do nothing, which sends you debugging the wrong layer.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →