⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All CI/CD & GitOps Interview Questions Scenario 176 of 187 in CI/CD & GitOps
Senior DevOps / SRE CI/CD Deployment Strategies & Ingress Top 10 Toughest
🎯 Target Role / Context: Senior Platform Engineer / SRE Incident Response

Q: A deployment succeeded according to your CI/CD pipeline, but users are still receiving responses from the old version. Explain exactly where you start debugging and how you isolate the failure.

Step-by-step triage workflow when CI/CD reports a successful release, but live client traffic remains routed to the previous application version.

#CI/CD #Kubernetes #Ingress #DNS #Service Mesh #ArgoCD #Traffic Routing #Top 10 Toughest #Toughest Questions #Production Scenarios
🎙️ Candidate Opening & Architectural Context
"When a deployment reports green in CI/CD while production ingress still delivers the prior version, the failure is almost never in container building—it lies at the traffic routing and state convergence layer. I troubleshoot systematically from the user edge backwards: CDN caching, DNS routing, Ingress/Service endpoint registration, Deployment replica status, and GitOps sync state."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's Enterprise GitOps with ArgoCD & Kubernetes Rollouts covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: When a deployment reports green in CI/CD while production ingress still delivers the prior version, the failure is almost never in container building—it lies at the traffic routing and state convergence layer. I troubleshoot systematically from the user edge backwards: CDN caching, DNS routing, Ingress/Service endpoint registration, Deployment replica status, and GitOps sync state.

1

Verify Edge CDN & Browser Caching

Inspect HTTP response headers directly from the edge using curl. Check 'Age', 'CF-Cache-Status', 'X-Cache', or 'Cache-Control' headers. Frequently, static assets (JS bundles, HTML index files) or API responses are aggressively cached at Cloudflare, CloudFront, or client service workers even though backend pods were replaced.

curl -I -s -H "Cache-Control: no-cache" https://api.example.com/v1/version
# Check response headers:
# x-cache: Hit from cloudfront
# age: 84320
2

Verify Ingress & Service Selector Endpoint Binding

Check whether the Kubernetes Service or Ingress is actually pointing to the new Pod replicas. In Blue/Green or Canary deployments (Argo Rollouts, Istio, Flagger), the new ReplicaSet may be healthy but the Service selector, Istio VirtualService weight, or ALB Target Group weight never flipped to active.

kubectl get endpoints <service-name> -o yaml
kubectl get svc <service-name> -o jsonpath='{.spec.selector}'
# For Argo Rollouts:
kubectl argo rollouts status rollout <app-name>
kubectl argo rollouts get rollout <app-name>
Advertisement
3

Check Pod Image Digest & Deployment Rollout Progress

Verify what image digest is actively running inside the pods. If developers used mutable tags (e.g. ':latest' or ':staging') with 'imagePullPolicy: IfNotPresent', the deployment updated its manifest, but the node continued running the cached old container image.

kubectl get deployment <app-name> -o jsonpath='{.spec.template.spec.containers[*].image}'
kubectl get pods -l app=<app-name> -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[*].imageID}{"\n"}{end}'
4

Validate GitOps Reconciliation & Pipeline Artifact Pinning

Confirm whether ArgoCD, Flux, or Jenkins deployed the expected commit SHA. Ensure the pipeline didn't overwrite the manifest in a detached branch or experience silent git push rejection.

Pro Tip: Golden Rule: Always deploy immutable image tags pinned to git commit SHAs (e.g. app:v1.14.2-a8f3b9c) and enforce imagePullPolicy: Always to prevent silent image caching discrepancies.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"A 'successful deployment' in CI/CD only means the API accepted the manifest. True release verification requires checking Edge cache -> Ingress weight -> Service endpoints -> Pod image sha256 digests."
⚡ 60-Second Elevator Pitch Talking Points
  • Bypass CDN caching with curl -I to check Edge X-Cache and Age headers.
  • Inspect Service selectors and Ingress target groups to ensure traffic routing weights flipped to the new ReplicaSet.
  • Query running pod image digests using kubectl get pods -o jsonpath to confirm nodes pulled the new sha256 binary rather than a cached tag.
  • Verify GitOps sync status in ArgoCD/Flux to rule out out-of-sync branches or silent manifest rollbacks.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Promoting application releases to production without an automated, canary-validated rollback mechanism when configuring Deployment Succeeded Traffic Pipelines that lack automated rollback mechanisms force engineering teams to scramble manual hotfixes during severe production regressions.
  2. Running concurrent deployment pipeline jobs without stage mutex locking on shared target environments when configuring Deployment Succeeded Traffic Concurrent pipeline runs on the same environment race to apply state changes, causing out-of-order commits and unpredictable deployment state.
  3. Omitting dependency lockfiles (package-lock.json, go.sum, Cargo.lock) from continuous integration builds when configuring Deployment Succeeded Traffic Omitting lockfiles allows upstream package updates to break builds unpredictably without any changes to application codebase.
  4. Failing to define dynamic cache invalidation keys for CI package managers and layer caches when configuring Deployment Succeeded Traffic Stale build caches mask underlying dependency incompatibilities or pull outdated libraries, while un-cached runs slow feedback loops.
Advertisement
Want more CI/CD & GitOps scenarios?
Explore our complete collection of scenario-based CI/CD & GitOps interview runbooks.
Browse All CI/CD & GitOps Questions →