⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 176 of 207 in Kubernetes
Senior DevOps / SRE Kubernetes Kubernetes Ingress & Service Triage Production Runbook
🎯 Target Role / Context: Senior SRE Production Incident Drill

Q: Your pods are Running, but users see 503 errors. Where do you debug?

End-to-end 9-step SRE triage sequence when pods show status 'Running', node is healthy, CPU/RAM are normal, but users receive HTTP 503 errors.

#Kubernetes #Service #Ingress #503 Error #Troubleshooting
🎙️ Candidate Opening & Architectural Context
"A pod in the 'Running' state does NOT equal application health. A 503 Service Unavailable returned by an Ingress controller or Load Balancer means traffic cannot be routed to any healthy backend. In production, troubleshoot across the complete request flow: User → DNS → Load Balancer → Ingress → Service → EndpointSlices → Pod Network Namespace → Application Runtime → Backing Dependencies."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: A pod in the 'Running' state does NOT equal application health. A 503 Service Unavailable returned by an Ingress controller or Load Balancer means traffic cannot be routed to any healthy backend. In production, troubleshoot across the complete request flow: User → DNS → Load Balancer → Ingress → Service → EndpointSlices → Pod Network Namespace → Application Runtime → Backing Dependencies.

1

Inspect Pod Runtime Status & Ready Column

Examine the READY column. If pods show 0/1, they are running their containers but failing their readiness probes, causing the kubelet to detach them from Service endpoints.

kubectl get pods -n production -o wide
kubectl describe pod <pod-name> -n production
2

Check Application Logs & Previous Restarts

Inspect container stdout/stderr. Always use --previous to check logs from recently terminated or crashed instances.

kubectl logs <pod-name> -n production --tail=100
kubectl logs <pod-name> -n production --previous
Advertisement
3

Check Recent Cluster Events for Probe Failures

Look for Unhealthy probe warnings, OOM warnings, or sandbox network failures in chronological order.

kubectl get events -n production --sort-by='.metadata.creationTimestamp' | tail -n 25
4

Check Service & EndpointSlices Registration

Verify whether pods are actually attached to the Service. If Endpoints shows , traffic drops immediately at the Service boundary.

kubectl get service payment-service -n production
kubectl get endpoints payment-service -n production
kubectl get endpointslices -l kubernetes.io/service-name=payment-service -n production
5

Check Ingress Controller & Cloud Load Balancer Rules

Ensure the Ingress path rules route to the exact Service name and port. Check cloud load balancer target group health.

kubectl describe ingress payment-ingress -n production
kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --tail=50
6

Validate Internal Application Health & Probes

Bypass Ingress and Service to test the application port directly inside the container namespace.

kubectl exec -it <pod-name> -n production -- curl -v http://localhost:8080/healthz
# Or port-forward locally:
kubectl port-forward <pod-name> 8080:8080 -n production
7

Check Configuration & Secret Values

Validate that ConfigMaps and Secrets are properly mounted and contain non-empty, valid environment variables.

kubectl get configmap,secret -n production -l app=payment
8

Check Downstream Dependencies (DB, Kafka, APIs)

Ensure the pod can reach external datastores, caches, and internal services without network policy blocks.

kubectl exec -it <pod-name> -n production -- nc -zv postgres.internal 5432
kubectl exec -it <pod-name> -n production -- nc -zv redis.internal 6379
9

Check Recent Changes & Rollout History

Correlate with recent deployments or git commits that may have introduced configuration bugs or schema mismatches.

kubectl rollout history deployment/payment-deployment -n production
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pod Running != Ready. A 503 error means the proxy has no healthy upstream endpoints. Triage via the 9-step SRE sequence: Pod status -> Logs -> Events -> Endpoints -> Ingress -> Healthz -> Config -> Dependencies -> Rollout history."
⚡ 60-Second Elevator Pitch Talking Points
  • Remember: Pod Running != Application Healthy. Always walk through the full request flow: User → DNS → LB → Ingress → Service → Pod → Dependencies.
  • First check 'kubectl get endpoints': if Endpoints is <none>, pods are detached due to failing readiness probes or label selector mismatches.
  • Verify the READY column: 0/1 means the container started, but the readiness probe is failing, shielding traffic.
  • Exec into the pod or port-forward to test the local healthz endpoint directly to isolate app bugs from network routing.
  • After applying the fix, verify: pod shows 1/1 Ready, endpoints populated, app returns HTTP 200, and downstream metrics normalize.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Overlooking CoreDNS query amplification caused by high ndots search path configurations when addressing unavailable Default ndots:5 triggers multiple sequential failed DNS lookups for external domains against cluster local search paths, overloading CoreDNS.
  2. Applying restrictive NetworkPolicies that block CoreDNS egress UDP/TCP port 53 lookups Blocking egress DNS traffic prevents pods from resolving internal cluster services or external database endpoints.
  3. Confusing Service exposure ports with container targetPort definitions in Kubernetes manifests Mismatched ports forward traffic to unlistening container ports, resulting in immediate connection refused errors.
  4. Ignoring MTU packet size mismatches between physical host interfaces and CNI overlay encapsulation Overlay encapsulation headers (VXLAN/Geneve) exceed standard MTU without proper clamping, causing silent packet drops on large TCP payloads.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →