Q: Someone applied a bad NetworkPolicy that's blocking all traffic in the cluster. How do you recover?
1. Identify the policy: kubectl get networkpolicy -A — list all NetworkPolicies across namespaces.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "When troubleshooting Kubernetes, I always follow a structured layered model: Pod status -> Events -> Logs -> Network. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix."
Initial Diagnostics & Root Cause Analysis
Prevention:
- Identify the policy:
kubectl get networkpolicy -A— list all NetworkPolicies across namespaces. - Delete the bad one:
kubectl delete networkpolicy.-n - Traffic should restore immediately after deletion (NetworkPolicy is applied in near-real-time by the CNI plugin).
- Always test NetworkPolicy changes in a staging namespace first.
Remediation & Permanent Safeguards
- Use
kubectl apply --dry-run=serverto validate. - For complex policies, use tools like Cilium's policy editor to visualize impact before applying.
- Identify the policy: kubectl get networkpolicy -A — list all NetworkPolicies across namespaces.
- Delete the bad one: kubectl delete networkpolicy -n .
- Traffic should restore immediately after deletion (NetworkPolicy is applied in near-real-time by ...
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
-
Configuring Service selector labels that mismatch underlying Pod manifest metadata when configuring Someone applied NetworkPolicy
A typo in
spec.selectorcauses the Service endpoints list (kubectl get endpoints) to remain empty, resulting in503 Service Unavailableor timeouts. -
Omitting explicit ingressClassName specifications when running multiple ingress controllers when configuring Someone applied NetworkPolicy
Without an explicit
ingressClassName, ingress resources are either ignored or picked up unpredictably by competing controllers. -
Depleting worker node subnet IP address allocations under VPC CNI or Azure CNI when configuring Someone applied NetworkPolicy
Direct VPC pod networking allocates real subnet IPs; failing to monitor IP pools causes new pods to stay stuck indefinitely in
ContainerCreating. - Attempting network triage without inspecting kubectl describe service and EndpointSlice resources when configuring Someone applied NetworkPolicy Verifying EndpointSlice population immediately determines whether a connectivity failure is caused by pod labeling, readiness probes, or routing.