Q: A deployment to production failed midway. Half the instances are running the new version, half the old. What do you do?
This is a partial deployment — dangerous state. The two versions may be incompatible for traffic.
#CI/CD #Then copy app code (changes every commit) #L2 #DevOps #Automation #Pipelines
🎙️ Candidate Opening & Architectural Context
""During a high-stakes release, we hit a similar deployment challenge and resolved it with automated safeguards. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix.""
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Initial Diagnostics & Root Cause Analysis
This is a partial deployment — dangerous state. The two versions may be incompatible for traffic.
- Immediate action — roll back by completing the rollback on remaining instances. Don't leave a mixed state.
- Check backward compatibility — did the new version require a DB schema change that the old version doesn't support? If yes, you may have a data problem to deal with.
- Blue/green preferred — this wouldn't happen with blue/green deployments. Traffic is only cut over after the full new environment is verified healthy.
2️⃣
Remediation & Permanent Safeguards
Prevention: use blue/green or canary deployments with an automated rollback trigger.
- Investigate root cause — why did the deployment fail midway? Disk space? Memory? Bad health check?
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Immediate action — roll back by completing the rollback on remaining instances. Don't leave a mixed state.."
⚡ 60-Second Elevator Pitch Talking Points
- Immediate action — roll back by completing the rollback on remaining instances. Don't leave a mix...
- Check backward compatibility — did the new version require a DB schema change that the old versio...
- Blue/green preferred — this wouldn't happen with blue/green deployments. Traffic is only cut over...
Advertisement