Q: In a canary deployment to production, half the traffic returns HTTP 502 Bad Gateway, while the other half succeeds. Walk us through your troubleshooting approach.

Inspect the Ingress, VirtualService, or Load Balancer configuration to confirm how the 50/50 split is allocated. Identify the exact Pod IP addresses belonging to the canary ReplicaSet.

# Identify Pods in canary ReplicaSet
kubectl get pods -l app=payment-service --show-labels
kubectl get endpoints payment-service-canary -o wide

Common causes of 502 on newly deployed canaries: 1. **Container Port Mismatch**: Service targetPort does not match the actual port exposed by the new container image (e.g. Service routes to 8080, but new Dockerfile exposes 8000). 2. **Immediate Crash on Connection**: The canary pod binds the socket but crashes on the first incoming request due to missing environment variables or secret misconfigurations. 3. **TLS Mismatch**: Ingress expects plaintext HTTP upstream, but canary was configured with HTTPS (or vice versa).

# Test canary pod directly from within the cluster bypassing ingress
kubectl exec -it <debug-pod> -- curl -v http://<canary-pod-ip>:<target-port>/healthz

Before deep-dive debugging, mitigate user impact immediately. Roll canary traffic back to 0% in Argo Rollouts or patch the Istio VirtualService weights.

# Rollback canary immediately via Argo Rollouts
kubectl argo rollouts abort payment-service
# Or update Istio VirtualService weight to 100% stable
kubectl patch virtualservice payment-service --type merge -p '{"spec":{"http":[{"route":[{"destination":{"host":"payment-stable"},"weight":100},{"destination":{"host":"payment-canary"},"weight":0}]}]}}'

Check the canary pod logs and previous container logs (`kubectl logs --previous`) to analyze application exceptions, uncaught startup errors, or missing runtime configurations.