⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All CI/CD & GitOps Interview Questions Scenario 2 of 187 in CI/CD & GitOps
Senior DevOps / SRE CI/CD Release Engineering Critical Incident
🎯 Target Role / Context: Release Engineering & High Availability

Q: Production deployment fails halfway — what's your rollback strategy?

Engineering strategy for recovering from halfway-failed deployments: rolling update halts, automated canary rollback, and protecting stateful databases.

#CI/CD #Rollback #Database Migrations #Canary #Blue-Green #SRE
🎙️ Candidate Opening & Architectural Context
"A deployment failing halfway means your system is in a hybrid state — some nodes or pods are running the new version while others run the old version. The strategy depends on deployment pattern and data compatibility."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's Enterprise GitOps with ArgoCD & Kubernetes Rollouts covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: A deployment failing halfway means your system is in a hybrid state — some nodes or pods are running the new version while others run the old version. The strategy depends on deployment pattern and data compatibility.

1️⃣

Stop Forward Rollout & Contain Traffic

Halt the release immediately before more instances degrade:

  • In Kubernetes: Rolling updates halt automatically if new pods fail readiness checks or crash (governed by maxUnavailable).
  • In Canary (Argo Rollouts / Flagger): Issue an abort command (kubectl argo rollouts abort <app>) to shift 100% traffic back to the stable baseline.
  • In AWS CodeDeploy / ECS: Click Stop and roll back deployment.
2️⃣

Execute Safe Rollback

Restore all instances to the previous known-good state:

  • Run kubectl rollout undo deployment/<app> to revert to previous ReplicaSet.
  • If Blue/Green: Traffic router never switched over, or can be instantly routed back to Blue target group with 0 rebuild time.
  • Verify target health: Ensure all healthy pods are receiving traffic and serving 200 OK.
Advertisement
3️⃣

Verify Database & Stateful Dependencies

The most dangerous aspect of partial failures:

  • Did a migration script execute before the application containers failed?
  • If code rolled back but DB schema remains in new state: verify application is backward-compatible.
  • Golden Rule: Never deploy breaking DB schema changes and application code in the same release. Always use the Expand/Contract (Parallel Run) pattern.
4️⃣

Automated Guardrails to Prevent Recurrence

Architectural improvements for future releases:

  • Add synthetic canary health analysis: evaluate Prometheus 5xx error rate and latency over 5-minute intervals before promoting each traffic step.
  • Add pre-deployment health checks and automated database rollback scripts.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Halt forward rollout immediately. Revert traffic to the stable baseline. Never couple breaking database migrations with application deployments — use Expand/Contract."
⚡ 60-Second Elevator Pitch Talking Points
  • Halt rollout immediately: Abort canary or rolling update to prevent further degradation.
  • Revert instances: Use 'kubectl rollout undo' or flip traffic router back to 100% stable version.
  • Check Database: Verify if migrations ran; ensure schema changes follow Expand/Contract so old code remains functional.
  • Validate: Confirm healthy targets in ALB and verify application error metrics stabilize.
  • Prevent: Introduce automated Canary analysis (Argo Rollouts/Flagger) with automated rollback thresholds.
🎬 60-Second Video Walkthrough
Watch the Animated Breakdown & Practice Live

Mastered this procedure? Subscribe to @naveedkumbhar01 on YouTube for daily 60-second SRE incident runbooks, Kubernetes triage shortcuts, and high-impact interview elevator pitches.

🔔 Subscribe on YouTube (Free) ▶ Watch Short (60s) ↗
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Promoting application releases to production without an automated, canary-validated rollback mechanism when configuring Deployment Fails Halfway Pipelines that lack automated rollback mechanisms force engineering teams to scramble manual hotfixes during severe production regressions.
  2. Running concurrent deployment pipeline jobs without stage mutex locking on shared target environments when configuring Deployment Fails Halfway Concurrent pipeline runs on the same environment race to apply state changes, causing out-of-order commits and unpredictable deployment state.
  3. Omitting dependency lockfiles (package-lock.json, go.sum, Cargo.lock) from continuous integration builds when configuring Deployment Fails Halfway Omitting lockfiles allows upstream package updates to break builds unpredictably without any changes to application codebase.
  4. Failing to define dynamic cache invalidation keys for CI package managers and layer caches when configuring Deployment Fails Halfway Stale build caches mask underlying dependency incompatibilities or pull outdated libraries, while un-cached runs slow feedback loops.
Advertisement
Want more CI/CD & GitOps scenarios?
Explore our complete collection of scenario-based CI/CD & GitOps interview runbooks.
Browse All CI/CD & GitOps Questions →