Q: A deployment to production failed midway. Half the instances are running the new version, half the old. What do you do?
This is a partial deployment — dangerous state. The two versions may be incompatible for traffic.
Want to master this scenario in a live sandbox? KodeKloud's Enterprise GitOps with ArgoCD & Kubernetes Rollouts covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "During a high-stakes release, we hit a similar deployment challenge and resolved it with automated safeguards. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix."
Initial Diagnostics & Root Cause Analysis
This is a partial deployment — dangerous state. The two versions may be incompatible for traffic.
- Immediate action — roll back by completing the rollback on remaining instances. Don't leave a mixed state.
- Check backward compatibility — did the new version require a DB schema change that the old version doesn't support? If yes, you may have a data problem to deal with.
- Blue/green preferred — this wouldn't happen with blue/green deployments. Traffic is only cut over after the full new environment is verified healthy.
Remediation & Permanent Safeguards
Prevention: use blue/green or canary deployments with an automated rollback trigger.
- Investigate root cause — why did the deployment fail midway? Disk space? Memory? Bad health check?
- Immediate action — roll back by completing the rollback on remaining instances. Don't leave a mix...
- Check backward compatibility — did the new version require a DB schema change that the old versio...
- Blue/green preferred — this wouldn't happen with blue/green deployments. Traffic is only cut over...
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Promoting application releases to production without an automated, canary-validated rollback mechanism when configuring deployment failed midway Pipelines that lack automated rollback mechanisms force engineering teams to scramble manual hotfixes during severe production regressions.
- Running concurrent deployment pipeline jobs without stage mutex locking on shared target environments when configuring deployment failed midway Concurrent pipeline runs on the same environment race to apply state changes, causing out-of-order commits and unpredictable deployment state.
- Omitting dependency lockfiles (package-lock.json, go.sum, Cargo.lock) from continuous integration builds when configuring deployment failed midway Omitting lockfiles allows upstream package updates to break builds unpredictably without any changes to application codebase.
- Failing to define dynamic cache invalidation keys for CI package managers and layer caches when configuring deployment failed midway Stale build caches mask underlying dependency incompatibilities or pull outdated libraries, while un-cached runs slow feedback loops.