Q: How do you perform a rollback of a failed deployment in your CI/CD pipeline? Walk me through your complete operational workflow from failure detection to post-incident prevention.
End-to-end operational guide for rolling back failed deployments in CI/CD pipelines, based on the Detect → Analyze → Rollback → Validate → Monitor → RCA lifecycle.
Want to master this scenario in a live sandbox? KodeKloud's Enterprise GitOps with ArgoCD & Kubernetes Rollouts covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Stage 1 & 2: Rapid Failure Detection & Scope Analysis
Detect anomalies via pipeline health checks and live monitoring: - Pipeline logs (Jenkins / GitHub Actions / GitLab CI) - Application error logs (Elasticsearch / Loki / Datadog) - Container status (`CrashLoopBackOff`, `OOMKilled`) - Alertmanager & PagerDuty error rate alerts Quickly identify the last known stable deployment artifact (Git commit SHA, Docker image digest, or Helm release revision).
# Identify previous stable Helm release or Kubernetes rollout revision
helm history my-app -n production
kubectl rollout history deployment/my-app -n productionStage 3: Execute Controlled Rollback
Trigger automated or manual rollback according to infrastructure target:
- **Kubernetes**: Run `kubectl rollout undo deployment/
# Fast rollback commands
kubectl rollout undo deployment/my-app -n production
# Or via Helm:
helm rollback my-app 42 -n productionStage 4 & 5: Post-Rollback Validation & Active Monitoring
Validate that the rolled-back revision is healthy: - Synthetic HTTP health checks (`/healthz` returning 200) - Pod container ready status (`kubectl get pods -l app=my-app`) - Error rate drops back below SLO thresholds in Grafana/Datadog - CPU and memory usage return to normal baselines
Stage 6 & 7: Root Cause Analysis (RCA) & Preventative Safeguards
Conduct blameless post-mortem answering four fundamental questions: 1. **What failed?** (Exact component or code line) 2. **Why did it fail?** (Configuration drift, schema mismatch, unhandled null pointer) 3. **Why wasn't it detected earlier?** (Missing integration test or staging disparity) 4. **What preventive action is required?** (Automated canary analysis, pre-flight DB checks, or contract tests)
- Execute an immediate rollback to the last verified image tag or Helm revision.
- Verify service restoration across synthetic health probes, pod statuses, and error rates.
- Maintain continuous monitoring post-rollback to ensure performance stabilization.
- Conduct blameless RCA addressing why the regression bypassed pre-deployment CI test gates.