Q: A request is going from Pod A to Pod B via a Service and it's very slow. How do you troubleshoot network latency in Kubernetes?
1. Baseline test — kubectl exec -it <pod-a> -- curl -o /dev/null -s -w "%{time_total}" http://<service>:<port> to measure actual latency.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "When troubleshooting Kubernetes, I always follow a structured layered model: Pod status -> Events -> Logs -> Network. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix."
Initial Diagnostics & Root Cause Analysis
- Baseline test —
kubectl exec -itto measure actual latency.-- curl -o /dev/null -s -w "%{time_total}" http:// : - Bypass the service — test pod-to-pod directly using the pod IP to see if latency is in kube-proxy/iptables:
kubectl exec -it.-- curl http:// : - Check kube-proxy mode — iptables vs ipvs. ipvs is faster at scale.
- Check CNI — network plugin issues. Run
pingbetween pods to test raw network latency.
Remediation & Permanent Safeguards
Execute the resolution runbook and verify workload health:
- DNS latency —
kubectl exec -it— DNS lookups through CoreDNS add latency. Consider-- time nslookup ndots:5setting impact. - Node-level network — check if nodes are on the same AZ. Cross-AZ traffic adds ~1-2ms.
- Baseline test — kubectl exec -it -- curl -o /dev/null -s -w "%{time_total}" http://: to measure ...
- Bypass the service — test pod-to-pod directly using the pod IP to see if latency is in kube-proxy...
- Check kube-proxy mode — iptables vs ipvs. ipvs is faster at scale.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
-
Configuring Service selector labels that mismatch underlying Pod manifest metadata when configuring request going Service
A typo in
spec.selectorcauses the Service endpoints list (kubectl get endpoints) to remain empty, resulting in503 Service Unavailableor timeouts. -
Omitting explicit ingressClassName specifications when running multiple ingress controllers when configuring request going Service
Without an explicit
ingressClassName, ingress resources are either ignored or picked up unpredictably by competing controllers. -
Depleting worker node subnet IP address allocations under VPC CNI or Azure CNI when configuring request going Service
Direct VPC pod networking allocates real subnet IPs; failing to monitor IP pools causes new pods to stay stuck indefinitely in
ContainerCreating. - Attempting network triage without inspecting kubectl describe service and EndpointSlice resources when configuring request going Service Verifying EndpointSlice population immediately determines whether a connectivity failure is caused by pod labeling, readiness probes, or routing.