Q: Your pods are Running, but users see 503 errors. Where do you debug?
End-to-end 9-step SRE triage sequence when pods show status 'Running', node is healthy, CPU/RAM are normal, but users receive HTTP 503 errors.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: A pod in the 'Running' state does NOT equal application health. A 503 Service Unavailable returned by an Ingress controller or Load Balancer means traffic cannot be routed to any healthy backend. In production, troubleshoot across the complete request flow: User → DNS → Load Balancer → Ingress → Service → EndpointSlices → Pod Network Namespace → Application Runtime → Backing Dependencies.
Inspect Pod Runtime Status & Ready Column
Examine the READY column. If pods show 0/1, they are running their containers but failing their readiness probes, causing the kubelet to detach them from Service endpoints.
kubectl get pods -n production -o wide
kubectl describe pod <pod-name> -n production
Check Application Logs & Previous Restarts
Inspect container stdout/stderr. Always use --previous to check logs from recently terminated or crashed instances.
kubectl logs <pod-name> -n production --tail=100
kubectl logs <pod-name> -n production --previous
Check Recent Cluster Events for Probe Failures
Look for Unhealthy probe warnings, OOM warnings, or sandbox network failures in chronological order.
kubectl get events -n production --sort-by='.metadata.creationTimestamp' | tail -n 25
Check Service & EndpointSlices Registration
Verify whether pods are actually attached to the Service. If Endpoints shows
kubectl get service payment-service -n production
kubectl get endpoints payment-service -n production
kubectl get endpointslices -l kubernetes.io/service-name=payment-service -n production
Check Ingress Controller & Cloud Load Balancer Rules
Ensure the Ingress path rules route to the exact Service name and port. Check cloud load balancer target group health.
kubectl describe ingress payment-ingress -n production
kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --tail=50
Validate Internal Application Health & Probes
Bypass Ingress and Service to test the application port directly inside the container namespace.
kubectl exec -it <pod-name> -n production -- curl -v http://localhost:8080/healthz
# Or port-forward locally:
kubectl port-forward <pod-name> 8080:8080 -n production
Check Configuration & Secret Values
Validate that ConfigMaps and Secrets are properly mounted and contain non-empty, valid environment variables.
kubectl get configmap,secret -n production -l app=payment
Check Downstream Dependencies (DB, Kafka, APIs)
Ensure the pod can reach external datastores, caches, and internal services without network policy blocks.
kubectl exec -it <pod-name> -n production -- nc -zv postgres.internal 5432
kubectl exec -it <pod-name> -n production -- nc -zv redis.internal 6379
Check Recent Changes & Rollout History
Correlate with recent deployments or git commits that may have introduced configuration bugs or schema mismatches.
kubectl rollout history deployment/payment-deployment -n production
- Remember: Pod Running != Application Healthy. Always walk through the full request flow: User → DNS → LB → Ingress → Service → Pod → Dependencies.
- First check 'kubectl get endpoints': if Endpoints is <none>, pods are detached due to failing readiness probes or label selector mismatches.
- Verify the READY column: 0/1 means the container started, but the readiness probe is failing, shielding traffic.
- Exec into the pod or port-forward to test the local healthz endpoint directly to isolate app bugs from network routing.
- After applying the fix, verify: pod shows 1/1 Ready, endpoints populated, app returns HTTP 200, and downstream metrics normalize.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
-
Overlooking CoreDNS query amplification caused by high ndots search path configurations when addressing unavailable
Default
ndots:5triggers multiple sequential failed DNS lookups for external domains against cluster local search paths, overloading CoreDNS. - Applying restrictive NetworkPolicies that block CoreDNS egress UDP/TCP port 53 lookups Blocking egress DNS traffic prevents pods from resolving internal cluster services or external database endpoints.
-
Confusing Service exposure ports with container targetPort definitions in Kubernetes manifests
Mismatched ports forward traffic to unlistening container ports, resulting in immediate
connection refusederrors. - Ignoring MTU packet size mismatches between physical host interfaces and CNI overlay encapsulation Overlay encapsulation headers (VXLAN/Geneve) exceed standard MTU without proper clamping, causing silent packet drops on large TCP payloads.