Q: Your pods are Running, but users see 503 errors. Where do you debug?
Root cause isolation when pods show status 'Running', but the Ingress controller returns HTTP 503 Service Unavailable to end users.
#Kubernetes #Service #Ingress #503 Error #Troubleshooting
🎙️ Candidate Opening & Architectural Context
"A 503 Service Unavailable returned by an Ingress controller means the Ingress cannot route traffic to any healthy backend. Even if pods are in the 'Running' state, they may be excluded from the Service's Endpoints due to failing readiness probes or label mismatches."
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1
Inspect Service Endpoints and EndpointSlices
Check if the Service has registered endpoints. If Endpoints shows
kubectl get endpoints payment-service -n production
kubectl get endpointslices -l kubernetes.io/service-name=payment-service -n production
2
Check Pod Readiness Probe vs. Liveness Probe
Pods can be in the 'Running' state while their readiness probe is failing. When readiness fails, the pod remains running, but the kubelet immediately unregisters its IP from the Service Endpoints.
kubectl get pods -l app=payment -n production -o wide
# Check the READY column: 0/1 means running but unready!
kubectl describe pod <pod-name> -n production | grep -A 5 "Readiness"
3
Verify Service Selector Label Mismatches
Compare service.spec.selector with pod.metadata.labels. A typo in version: v2.1 in the deployment while the service selects version: v2.0 will orphan all pods.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Running != Ready. If Endpoints is <none>, check readiness probe failures (0/1 Ready) or service selector label mismatches."
⚡ 60-Second Elevator Pitch Talking Points
- Check kubectl get endpoints: verify if the Service actually contains target pod IP addresses.
- Inspect the READY column in kubectl get pods: 0/1 means the pod is Running but failing its readinessProbe.
- Verify Service selector labels: ensure they match deployment pod template labels exactly.
- Check Ingress upstream timeouts and target port matching in the Service definition.
Advertisement