Q: A production deployment failed halfway through, leaving different service versions running. How would you recover with minimal customer impact?
Disaster recovery runbook for handling a mid-flight deployment failure that leaves inconsistent service versions running in production, handling database backward-compatibility and zero customer impact.
Want to master this scenario in a live sandbox? KodeKloud's Enterprise GitOps with ArgoCD & Kubernetes Rollouts covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: A split-version state in production poses high risk if the new version introduced contract changes or database schema updates. My response depends on whether the database schema is backward-compatible and whether the failure is transient or functional.
Freeze Deployment Pipeline & Stop Further Rollout
Immediately pause the deployment to stop the orchestrator from provisioning more broken replicas or killing remaining healthy legacy replicas.
# Pause Kubernetes deployment rollout
kubectl rollout pause deployment/payment-api -n production
Evaluate Backward Compatibility & Schema State
Check if database migrations executed. If migrations followed the Expand/Contract (Parallel Run) pattern, old and new application versions can safely run concurrently against the database without data corruption.
Execute Fast Rollback or Traffic Pinning
If the new version is throwing 5xx errors, execute an immediate rollback to the previous revision. If using Ingress/Service Mesh, route 100% of user traffic to the stable v1 subset.
kubectl rollout undo deployment/payment-api -n production
kubectl rollout status deployment/payment-api -n production
- Immediately pause the rollout (kubectl rollout pause) to prevent further replica destruction.
- Verify database backward compatibility: ensure migrations follow the Expand/Contract pattern.
- Issue an instant rollback (kubectl rollout undo) to restore 100% of traffic to the stable version.
- Review post-mortem: enforce pre-flight smoke tests, automated rollback triggers, and Blue/Green deployment.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Promoting application releases to production without an automated, canary-validated rollback mechanism when configuring Deployment Fails Halfway Pipelines that lack automated rollback mechanisms force engineering teams to scramble manual hotfixes during severe production regressions.
- Running concurrent deployment pipeline jobs without stage mutex locking on shared target environments when configuring Deployment Fails Halfway Concurrent pipeline runs on the same environment race to apply state changes, causing out-of-order commits and unpredictable deployment state.
- Omitting dependency lockfiles (package-lock.json, go.sum, Cargo.lock) from continuous integration builds when configuring Deployment Fails Halfway Omitting lockfiles allows upstream package updates to break builds unpredictably without any changes to application codebase.
- Failing to define dynamic cache invalidation keys for CI package managers and layer caches when configuring Deployment Fails Halfway in barclays-q004-partial-deployment-failure-split-versions Stale build caches mask underlying dependency incompatibilities or pull outdated libraries, while un-cached runs slow feedback loops.