Q: Someone applied a bad NetworkPolicy that's blocking all traffic in the cluster. How do you recover?
1. Identify the policy: kubectl get networkpolicy -A — list all NetworkPolicies across namespaces.
#Kubernetes #Advanced Scenarios #L2 #Container Orchestration #K8s
🎙️ Candidate Opening & Architectural Context
""When troubleshooting Kubernetes, I always follow a structured layered model: Pod status -> Events -> Logs -> Network. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix.""
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Initial Diagnostics & Root Cause Analysis
Prevention:
- Identify the policy:
kubectl get networkpolicy -A— list all NetworkPolicies across namespaces. - Delete the bad one:
kubectl delete networkpolicy.-n - Traffic should restore immediately after deletion (NetworkPolicy is applied in near-real-time by the CNI plugin).
- Always test NetworkPolicy changes in a staging namespace first.
2️⃣
Remediation & Permanent Safeguards
- Use
kubectl apply --dry-run=serverto validate. - For complex policies, use tools like Cilium's policy editor to visualize impact before applying.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Identify the policy: kubectl get networkpolicy -A — list all NetworkPolicies across namespaces.."
⚡ 60-Second Elevator Pitch Talking Points
- Identify the policy: kubectl get networkpolicy -A — list all NetworkPolicies across namespaces.
- Delete the bad one: kubectl delete networkpolicy -n .
- Traffic should restore immediately after deletion (NetworkPolicy is applied in near-real-time by ...
Advertisement