Q: A deployment succeeded according to your CI/CD pipeline, but users are still receiving responses from the old version. Explain exactly where you start debugging and how you isolate the failure.
Step-by-step triage workflow when CI/CD reports a successful release, but live client traffic remains routed to the previous application version.
Want to master this scenario in a live sandbox? KodeKloud's Enterprise GitOps with ArgoCD & Kubernetes Rollouts covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Verify Edge CDN & Browser Caching
Inspect HTTP response headers directly from the edge using curl. Check 'Age', 'CF-Cache-Status', 'X-Cache', or 'Cache-Control' headers. Frequently, static assets (JS bundles, HTML index files) or API responses are aggressively cached at Cloudflare, CloudFront, or client service workers even though backend pods were replaced.
curl -I -s -H "Cache-Control: no-cache" https://api.example.com/v1/version
# Check response headers:
# x-cache: Hit from cloudfront
# age: 84320Verify Ingress & Service Selector Endpoint Binding
Check whether the Kubernetes Service or Ingress is actually pointing to the new Pod replicas. In Blue/Green or Canary deployments (Argo Rollouts, Istio, Flagger), the new ReplicaSet may be healthy but the Service selector, Istio VirtualService weight, or ALB Target Group weight never flipped to active.
kubectl get endpoints <service-name> -o yaml
kubectl get svc <service-name> -o jsonpath='{.spec.selector}'
# For Argo Rollouts:
kubectl argo rollouts status rollout <app-name>
kubectl argo rollouts get rollout <app-name>Check Pod Image Digest & Deployment Rollout Progress
Verify what image digest is actively running inside the pods. If developers used mutable tags (e.g. ':latest' or ':staging') with 'imagePullPolicy: IfNotPresent', the deployment updated its manifest, but the node continued running the cached old container image.
kubectl get deployment <app-name> -o jsonpath='{.spec.template.spec.containers[*].image}'
kubectl get pods -l app=<app-name> -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[*].imageID}{"\n"}{end}'Validate GitOps Reconciliation & Pipeline Artifact Pinning
Confirm whether ArgoCD, Flux, or Jenkins deployed the expected commit SHA. Ensure the pipeline didn't overwrite the manifest in a detached branch or experience silent git push rejection.
- Bypass CDN caching with curl -I to check Edge X-Cache and Age headers.
- Inspect Service selectors and Ingress target groups to ensure traffic routing weights flipped to the new ReplicaSet.
- Query running pod image digests using kubectl get pods -o jsonpath to confirm nodes pulled the new sha256 binary rather than a cached tag.
- Verify GitOps sync status in ArgoCD/Flux to rule out out-of-sync branches or silent manifest rollbacks.