Q: In a canary deployment to production, half the traffic returns HTTP 502 Bad Gateway, while the other half succeeds. Walk us through your troubleshooting approach.
Diagnostic framework for resolving split-traffic 502 Bad Gateway errors during a canary deployment rollout in Kubernetes.
Want to master this scenario in a live sandbox? KodeKloud's Istio Service Mesh & Advanced Kubernetes Networking Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: When exactly 50% of requests return HTTP 502 during a canary deployment while the remaining 50% succeed, traffic is splitting evenly between two upstream target groups: one healthy (stable version) and one failing (canary version). A 502 indicates that the Ingress controller or Service Mesh reverse proxy received an invalid response or connection reset (RST) from the canary backend.
Confirm Traffic Split Routing & Upstream Targets
Inspect the Ingress, VirtualService, or Load Balancer configuration to confirm how the 50/50 split is allocated. Identify the exact Pod IP addresses belonging to the canary ReplicaSet.
# Identify Pods in canary ReplicaSet
kubectl get pods -l app=payment-service --show-labels
kubectl get endpoints payment-service-canary -o wide
Inspect Upstream Connection Resets & Port Mismatches
Common causes of 502 on newly deployed canaries: 1. Container Port Mismatch: Service targetPort does not match the actual port exposed by the new container image (e.g. Service routes to 8080, but new Dockerfile exposes 8000). 2. Immediate Crash on Connection: The canary pod binds the socket but crashes on the first incoming request due to missing environment variables or secret misconfigurations. 3. TLS Mismatch: Ingress expects plaintext HTTP upstream, but canary was configured with HTTPS (or vice versa).
# Test canary pod directly from within the cluster bypassing ingress
kubectl exec -it <debug-pod> -- curl -v http://<canary-pod-ip>:<target-port>/healthz
Halt Canary Progression & Route Traffic Back to 100% Stable
Before deep-dive debugging, mitigate user impact immediately. Roll canary traffic back to 0% in Argo Rollouts or patch the Istio VirtualService weights.
# Rollback canary immediately via Argo Rollouts
kubectl argo rollouts abort payment-service
# Or update Istio VirtualService weight to 100% stable
kubectl patch virtualservice payment-service --type merge -p '{"spec":{"http":[{"route":[{"destination":{"host":"payment-stable"},"weight":100},{"destination":{"host":"payment-canary"},"weight":0}]}]}}'
Extract Canary Logs & Core Dump
Check the canary pod logs and previous container logs (kubectl logs ) to analyze application exceptions, uncaught startup errors, or missing runtime configurations.
- Recognize that a 50% failure rate maps directly to the canary replica pool failing completely.
- Immediately abort the canary rollout to direct 100% of user traffic back to the stable deployment.
- Verify container targetPort mapping against the Kubernetes Service spec.
- Curl the canary pod IP directly from a cluster debug pod to isolate whether the app is crashing or misconfigured.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Confusing ALB 502 Bad Gateway (target connection reset) with 504 Gateway Timeout (upstream response timeout) when configuring Canary Deployment Returns 502 indicates the backend target closed the TCP connection prematurely or returned malformed HTTP, whereas 504 means the target failed to respond before idle timeout.
- Setting target deregistration delay to zero instead of allowing connection draining when configuring Canary Deployment Returns Instant target deregistration severs in-flight HTTP requests during auto-scaling scale-in or rolling deployments, causing immediate user-facing drops.
-
Failing to query CloudWatch ELB metrics (
HTTPCode_ELB_5XXvsHTTPCode_Target_5XX) during load balancer outages when configuring Canary Deployment Returns Comparing load balancer generated errors against target generated errors immediately isolates whether the issue lies in the proxy tier or application runtime. - Overlooking Security Group rules on backend EC2/ECS instances that block health check traffic from the ALB when configuring Canary Deployment Returns Target instances must explicitly permit inbound HTTP/HTTPS traffic from the ALB security group on the configured container/application port.