Q: You deployed a Cilium update, and suddenly inter-pod communication breaks for one namespace. What’s your triage approach to confirm root cause?
Production triage runbook when a Cilium eBPF CNI update breaks pod-to-pod network traffic selectively across a single namespace.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Use Cilium CLI to Inspect Namespace Endpoints & Identity Maps
Run `cilium-dbg endpoint list` on the node hosting affected pods. Verify whether pods in the broken namespace have valid Cilium Security Identities or are flagged in `WaitingForIdentity` or `Disconnecting` states.
# Inspect endpoints and security identities
cilium-dbg endpoint list
# Or via Kubernetes CLI:
kubectl get ciliumendpoints -n <broken-namespace>Audit NetworkPolicies & CiliumNetworkPolicies in the Namespace
Cilium updates frequently introduce stricter parser validations for `CiliumNetworkPolicy` or `CiliumClusterwideNetworkPolicy`. An existing policy with deprecated selectors or malformed port definitions may suddenly compile into a default-deny BPF filter.
kubectl get cnp,netpol -n <broken-namespace>
# Temporarily inspect policy drop events via Cilium monitor
cilium-dbg monitor --type dropInspect BPF Map Drops & Packet Trace with Hubble
Leverage **Hubble** (Cilium's eBPF observability engine) to trace dropped packets in real time. Hubble displays the exact dropped TCP flow, the source/destination pod identity, and the exact policy rule verdict.
hubble observe --namespace <broken-namespace> --verdict DROPPED
# Output pattern:
# TIMESTAMP: default/pod-a -> broken-ns/pod-b: DROPPED (Policy denied by cnp: deny-all-ingress)Re-synchronize Cilium Endpoint BPF Maps or Roll Back Policy
If BPF maps are corrupted or out of sync with etcd, regenerate the endpoint BPF programs without restarting pods.
cilium-dbg endpoint regenerate <endpoint-id>- Use hubble observe --verdict DROPPED to see real-time packet drops and the exact policy rule responsible.
- Check cilium-dbg endpoint list to ensure pods have assigned security identities.
- Review CiliumNetworkPolicies in the namespace for newly enforced syntax rules.
- Regenerate BPF endpoint maps via cilium-dbg endpoint regenerate to restore connectivity.