Q: Your CI/CD pipeline deploys directly to production on every merge to `main`. One Friday afternoon, a developer merges a PR and goes offline. The deployment silently fails halfway through — 30% of pods are running new code, 70% are running old code. There are no automated rollback hooks. How do you recover, and how do you prevent this situation architecturally?
Immediate recovery:
Want to master this scenario in a live sandbox? KodeKloud's Enterprise GitOps with ArgoCD & Kubernetes Rollouts covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "In enterprise CI/CD, you cannot rely on manual interventions; every rollback and promotion must be declarative. The interviewer is testing: Deployment readiness, automated rollback, on-call accountability.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Initial Diagnostics & Root Cause Analysis
Immediate recovery:
- Identify current state:
kubectl rollout status deployment/api— ifProgressingis stuck, the rollout is hanging. - Manually roll back:
kubectl rollout undo deployment/api— Kubernetes reverts to the previous ReplicaSet revision immediately. Verify withkubectl rollout history deployment/api. --atomicon Helm, or rollout analysis on Argo Rollouts — if the deployment does not reach 100% healthy within a timeout, it automatically rolls back.
Remediation & Permanent Safeguards
Architectural prevention:
- Deployment freeze rules: Block merges to
mainafter 3 PM on Fridays via a GitHub branch protection check that queries the current time. Simple but highly effective. - Require on-call acknowledgement: The pipeline sends a Slack message with "Deployment starting, confirm you're watching" and waits 60 seconds for a reaction emoji before proceeding. If no reaction, the pipeline pauses and pages on-call.
- Identify current state: kubectl rollout status deployment/api — if Progressing is stuck, the roll...
- Manually roll back: kubectl rollout undo deployment/api — Kubernetes reverts to the previous Repl...
- --atomic on Helm, or rollout analysis on Argo Rollouts — if the deployment does not reach 100% he...
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Promoting application releases to production without an automated, canary-validated rollback mechanism when configuring CI/CD pipeline deploys Pipelines that lack automated rollback mechanisms force engineering teams to scramble manual hotfixes during severe production regressions.
- Running concurrent deployment pipeline jobs without stage mutex locking on shared target environments when configuring CI/CD pipeline deploys Concurrent pipeline runs on the same environment race to apply state changes, causing out-of-order commits and unpredictable deployment state.
- Omitting dependency lockfiles (package-lock.json, go.sum, Cargo.lock) from continuous integration builds when configuring CI/CD pipeline deploys Omitting lockfiles allows upstream package updates to break builds unpredictably without any changes to application codebase.
- Failing to define dynamic cache invalidation keys for CI package managers and layer caches when configuring CI/CD pipeline deploys in ci-cd-q088-your-ci-cd-pipeline-deploys-directl Stale build caches mask underlying dependency incompatibilities or pull outdated libraries, while un-cached runs slow feedback loops.