Q: During high traffic, your app shows intermittent 502 errors through Ingress — how do you debug and fix it?
Root-cause analysis and resolution for intermittent HTTP 502 Bad Gateway errors during traffic peaks: diagnosing upstream timeouts, CPU throttling, readiness probe flapping, tuning Ingress proxy timeouts, and autoscaling HPA.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: I debug 502s from the edge inward: Ingress controller logs and metrics, upstream service endpoints, pod readiness, connection saturation, timeouts, and application logs. Under high traffic, common causes are insufficient replicas, slow upstreams, readiness flapping, keepalive/timeout mismatches, or node CPU throttling. I correlate timestamps across Ingress, service, and application telemetry before tuning.
Ingress Controller Logs & Service Endpoints Validation
Differentiate whether the 502 is generated by the Ingress controller or returned by the backend application:
# Filter Ingress controller logs for 502 responses
kubectl logs -n ingress-nginx deploy/ingress-nginx-controller --since=15m | grep ' 502 '
kubectl describe ingress public-api -n app
# Check service endpoints and watch pod restarts
kubectl get svc,endpoints -n app api -o wide
kubectl get pods -n app -l app=api -w
kubectl describe pod -n app <pod-name> | egrep 'Readiness|Liveness|Restart'
- Ingress Controller Error Logs: Search NGINX Ingress controller logs for
upstream timed out (110: Connection timed out)orconnect() failed (111: Connection refused). - Service Endpoints Check: Verify whether active endpoints are dropping or flapping under load.
- Pod Readiness Flapping: Check if high CPU causes readiness probes to time out, removing pods from endpoints dynamically.
Tuning Resource Pressure, HPA, and Upstream Proxy Timeouts
Remediate upstream latency bottlenecks and adjust Ingress controller keepalive and timeout limits:
# Inspect resource consumption and autoscaling
kubectl top pods -n app
kubectl top nodes
kubectl get hpa -n app
kubectl describe hpa api -n app
# Extend NGINX Ingress timeout annotations
kubectl annotate ingress public-api -n app nginx.ingress.kubernetes.io/proxy-read-timeout='120' --overwrite
kubectl annotate ingress public-api -n app nginx.ingress.kubernetes.io/proxy-send-timeout='120' --overwrite
- CPU Throttling & HPA: Inspect
kubectl top podsand HPA metrics; increase HPA minReplicas so capacity is pre-warmed for peaks. - Tune Proxy Timeouts: If backend queries take longer during traffic spikes, annotate the Ingress to extend proxy read and send timeouts from default 60s to 120s.
- Database & Query Optimization: Resolve backend bottleneck (e.g. unindexed query or connection pool starvation) that triggered slow upstream processing.
- Inspect NGINX Ingress controller logs to verify whether errors are upstream connection timeouts or dropped endpoints.
- Check pod CPU throttling and readiness probe failures that temporarily remove pods from Service endpoints during traffic spikes.
- Apply permanent fixes: increase HPA minReplicas, optimize upstream connection pools, and tune Ingress proxy-read-timeout annotations.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Confusing ALB 502 Bad Gateway (target connection reset) with 504 Gateway Timeout (upstream response timeout) when configuring Kubernetes Ingress Gateway 502 indicates the backend target closed the TCP connection prematurely or returned malformed HTTP, whereas 504 means the target failed to respond before idle timeout.
- Setting target deregistration delay to zero instead of allowing connection draining when configuring Kubernetes Ingress Gateway Instant target deregistration severs in-flight HTTP requests during auto-scaling scale-in or rolling deployments, causing immediate user-facing drops.
-
Failing to query CloudWatch ELB metrics (
HTTPCode_ELB_5XXvsHTTPCode_Target_5XX) during load balancer outages when configuring Kubernetes Ingress Gateway Comparing load balancer generated errors against target generated errors immediately isolates whether the issue lies in the proxy tier or application runtime. - Overlooking Security Group rules on backend EC2/ECS instances that block health check traffic from the ALB when configuring Kubernetes Ingress Gateway Target instances must explicitly permit inbound HTTP/HTTPS traffic from the ALB security group on the configured container/application port.