Q: Two pods can't communicate even though NetworkPolicy allows it.
CNI plugin may not support NetworkPolicy. Check CNI (Flannel doesn't, Calico does).
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: When <strong>two pods can't communicate even though NetworkPolicy allows it</strong>, the rule you are staring at is rarely the problem. Check that your CNI enforces policies at all, that the source pod has a matching egress rule if egress is restricted, that DNS is allowed, that selectors mean AND not OR, and that the port is the pod port, not the Service port. Interviewers ask this because NetworkPolicy semantics are additive, directional and easy to misread, and debugging needs a methodical approach.
Mental Model: How NetworkPolicy Really Decides
A connection needs permission on both ends, and policies only restrict pods they select.
- Selection: a pod is isolated for ingress or egress only if some policy selects it for that direction. Otherwise it allows everything.
- Both directions: if the source pod is egress-restricted, you need an egress rule on the source and an ingress rule on the destination. Allowing only one side blocks traffic.
- Additive: policies are unions of allows; there are no deny rules in the core API, so a stray default-deny policy in the namespace can be the cause.
- Enforcement: the API server accepts NetworkPolicy objects regardless; only a CNI that implements them (Calico, Cilium, Antrea and others) enforces them. Plain Flannel does not.
- Stateful tracking: established return traffic is allowed automatically by the CNI, so you only describe who may initiate the connection, in each direction where isolation applies.
Hands-on: Pods Can't Communicate Even Though NetworkPolicy Allows It
Reproduce the failure from the client pod, then list every policy that selects either pod.
kubectl get netpol -A
kubectl describe netpol -n prod
kubectl get pods -n prod --show-labels
kubectl get ns --show-labels | grep -E 'prod|staging'
# Test from the client pod's network namespace
kubectl debug -it -n prod pod/client --image=nicolaka/netshoot --target=client -- \
sh -c 'nslookup backend.prod.svc.cluster.local; nc -zv backend.prod.svc.cluster.local 8080'
- List policies:
kubectl get netpol -Aandkubectl describe netpol -n prodshow which pods each policy selects and what it allows. - Check labels:
kubectl get pods -n prod --show-labelsandkubectl get ns --show-labels; policies match labels, and a typo makes a rule select nothing. - Test with the right tool: an ephemeral debug container avoids changing the app image.
The Usual Culprits: Why Allowed Traffic Still Fails
These account for most real cases.
# Correct AND semantics: pods labelled app=client in namespaces labelled team=payments
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: payments
podSelector:
matchLabels:
app: client
ports:
- protocol: TCP
port: 8080
- Missing egress or DNS: a default-deny egress policy blocks DNS to CoreDNS in
kube-system(UDP and TCP 53), so names fail even if the target IP is allowed. - AND versus OR in from/to: a single list item containing both
namespaceSelectorandpodSelectormeans AND; two separate dash items mean OR. Mixing them up opens too much or too little. - Namespace labels: a
namespaceSelectorneeds a label on the namespace itself. Kubernetes setskubernetes.io/metadata.nameautomatically, which is the reliable one to use. - Wrong port: policies match the destination pod's container port, not the Service port. If the Service maps 80 to 8080, allow 8080.
- Not the CNI you think: check which plugin is installed with
kubectl get pods -n kube-system; a non-enforcing plugin means policies are ignored and the real block is elsewhere (Service selector, readiness, mesh).
Verification with CNI Flow Tools
Ask the dataplane what it dropped instead of reasoning from YAML alone.
- Cilium:
hubble observe --namespace prod --verdict DROPPEDshows dropped flows and the policy reason;cilium monitor --type dropworks on the node. - Calico: enable policy logging with a Log action in a staged policy, or inspect with
calicoctland the node's iptables counters. - Control test: in a non-production namespace, remove one policy at a time to find the culprit, then restore it.
- Pod IP versus Service: test the pod IP directly; if that works and the Service name fails, the issue is the Service, DNS or kube-proxy, not NetworkPolicy.
- Host and probe traffic: kubelet health probes and hostNetwork pods are treated specially by many CNIs, so a policy that looks correct can still behave differently for node-originated traffic.
Production Practice
Make policy changes testable and reversible.
- Default deny plus explicit allows: include DNS egress in the baseline policy so teams do not break name resolution.
- Test in CI: tools such as
cyclonusor Cilium connectivity tests validate that policies behave as intended in a kind cluster. - Observability: keep flow logs (Hubble or equivalent) enabled so a dropped connection is a query, not a guess.
- Review checklist: for every new policy ask which pods it selects, which direction it covers, whether DNS is allowed, and what a test connection from an unauthorised pod should show.
- NetworkPolicy is directional and additive, and only a CNI that implements it enforces it.
- I list every policy selecting either pod, check egress and DNS, verify label and port matches, then test with a debug container.
- I confirm drops with Hubble or Calico logs and bake a DNS-allowing default-deny baseline into CI-tested policies.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Only writing an ingress rule on the destination If the source is egress-restricted, its own egress policy still blocks the connection.
- Putting namespaceSelector and podSelector in separate list items That is OR, not AND, so the rule allows far more than intended or something else than expected.
- Using the Service port in the policy Policies are evaluated against the pod's target port after Service translation.
- Assuming the cluster enforces NetworkPolicy On a CNI without policy support the objects are accepted but do nothing, which sends you debugging the wrong layer.