⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All CI/CD & GitOps Interview Questions Scenario 120 of 187 in CI/CD & GitOps
Senior DevOps / SRE CI/CD Incident Rollback & Deployment Strategies Barclays Classic

Q: A production deployment failed halfway through, leaving different service versions running. How would you recover with minimal customer impact?

Disaster recovery runbook for handling a mid-flight deployment failure that leaves inconsistent service versions running in production, handling database backward-compatibility and zero customer impact.

#Kubernetes #Rollout #Rollback #Canary #Barclays #CI/CD
🎙️ Candidate Opening & Architectural Context
"A split-version state in production poses high risk if the new version introduced contract changes or database schema updates. My response depends on whether the database schema is backward-compatible and whether the failure is transient or functional."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's Enterprise GitOps with ArgoCD & Kubernetes Rollouts covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: A split-version state in production poses high risk if the new version introduced contract changes or database schema updates. My response depends on whether the database schema is backward-compatible and whether the failure is transient or functional.

1

Freeze Deployment Pipeline & Stop Further Rollout

Immediately pause the deployment to stop the orchestrator from provisioning more broken replicas or killing remaining healthy legacy replicas.

# Pause Kubernetes deployment rollout
kubectl rollout pause deployment/payment-api -n production
2

Evaluate Backward Compatibility & Schema State

Check if database migrations executed. If migrations followed the Expand/Contract (Parallel Run) pattern, old and new application versions can safely run concurrently against the database without data corruption.

Advertisement
3

Execute Fast Rollback or Traffic Pinning

If the new version is throwing 5xx errors, execute an immediate rollback to the previous revision. If using Ingress/Service Mesh, route 100% of user traffic to the stable v1 subset.

kubectl rollout undo deployment/payment-api -n production
kubectl rollout status deployment/payment-api -n production
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Enforce the Expand/Contract database pattern so dual versions can safely co-exist. When a rollout stalls, pause immediately to stabilize, assess schema safety, and issue kubectl rollout undo."
⚡ 60-Second Elevator Pitch Talking Points
  • Immediately pause the rollout (kubectl rollout pause) to prevent further replica destruction.
  • Verify database backward compatibility: ensure migrations follow the Expand/Contract pattern.
  • Issue an instant rollback (kubectl rollout undo) to restore 100% of traffic to the stable version.
  • Review post-mortem: enforce pre-flight smoke tests, automated rollback triggers, and Blue/Green deployment.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Promoting application releases to production without an automated, canary-validated rollback mechanism when configuring Deployment Fails Halfway Pipelines that lack automated rollback mechanisms force engineering teams to scramble manual hotfixes during severe production regressions.
  2. Running concurrent deployment pipeline jobs without stage mutex locking on shared target environments when configuring Deployment Fails Halfway Concurrent pipeline runs on the same environment race to apply state changes, causing out-of-order commits and unpredictable deployment state.
  3. Omitting dependency lockfiles (package-lock.json, go.sum, Cargo.lock) from continuous integration builds when configuring Deployment Fails Halfway Omitting lockfiles allows upstream package updates to break builds unpredictably without any changes to application codebase.
  4. Failing to define dynamic cache invalidation keys for CI package managers and layer caches when configuring Deployment Fails Halfway in barclays-q004-partial-deployment-failure-split-versions Stale build caches mask underlying dependency incompatibilities or pull outdated libraries, while un-cached runs slow feedback loops.
Advertisement
Want more CI/CD & GitOps scenarios?
Explore our complete collection of scenario-based CI/CD & GitOps interview runbooks.
Browse All CI/CD & GitOps Questions →