Q: How do you decide between a rollback vs a hotfix during a high-severity production incident?
Structured SRE decision framework for choosing between an immediate rollback versus a forward hotfix during high-severity production incidents: evaluating blast radius, stateful database migration risks, feature flag killswitches, and MTTR.
Want to master this scenario in a live sandbox? KodeKloud's Enterprise GitOps with ArgoCD & Kubernetes Rollouts covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: During a Sev-1 production incident, my golden rule is: optimize for Mean Time to Recovery (MTTR) first, not for root cause analysis. I default to an immediate rollback when the incident is clearly correlated with a recent release, the change is stateless, and rollback is low-risk and well-understood. Conversely, I choose a forward hotfix or feature-flag mitigation only when an irreversible database migration has already run, when data formats have diverged, or when rolling back would cause greater customer data loss.
The SRE Incident Decision Matrix: Rollback vs Hotfix
Four critical criteria evaluated by the Incident Commander:
# Fast rollback in Kubernetes
kubectl rollout history deployment/api -n prod
kubectl rollout undo deployment/api -n prod
kubectl rollout status deployment/api -n prod --timeout=120s
# Fast feature flag mitigation via ConfigMap or environment variable
kubectl set env deployment/api FEATURE_NEW_PAYMENTS=false -n prod
- 1. Correlation to Recent Change: Did error rates or latency spike immediately following a deployment? If yes, rollback is the primary candidate.
- 2. Data & Schema Compatibility: Has a non-backward-compatible database migration executed? If rolling back app code will cause crashes on newly formatted DB tables, rollback is dangerous — forward hotfix is required.
- 3. Feature Flag Killswitch: Can the broken feature be deactivated in seconds via LaunchDarkly or ConfigMap toggle? If yes, toggle the flag before executing any code deployment.
- 4. Estimated MTTR: Rollback takes <3 minutes via
kubectl rollout undoor Helm. A code hotfix takes 20-40 minutes (writing code, PR review, CI pipeline run). If rollback is safe, never wait on a hotfix.
Incident Commander Execution & Verification Protocol
Maintaining command discipline and single-threaded ownership during the outage:
# Verify error rate recovery following rollback
kubectl logs deployment/api -n prod --since=5m | grep -iE 'error|exception' | wc -l
kubectl top pods -n prod -l app=api
- Single Recovery Owner: The Incident Commander assigns one engineer to execute the rollback/hotfix while other engineers gather diagnostic logs and communicate with stakeholders.
- No Concurrent Changes: Never execute multiple emergency config changes simultaneously, as this obscures the root cause and creates conflicting failure modes.
- Metric Verification: Confirm recovery using user-facing error rate metrics in Grafana before declaring incident mitigation.
- Default to immediate rollback for stateless deployments where recovery takes under 3 minutes.
- Use feature-flag killswitches to disable broken code paths instantly without redeploying containers.
- Choose a forward hotfix only when irreversible database migrations or data transformations prevent a safe rollback.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Promoting application releases to production without an automated, canary-validated rollback mechanism when configuring Incident Decision Framework Pipelines that lack automated rollback mechanisms force engineering teams to scramble manual hotfixes during severe production regressions.
- Running concurrent deployment pipeline jobs without stage mutex locking on shared target environments when configuring Incident Decision Framework Concurrent pipeline runs on the same environment race to apply state changes, causing out-of-order commits and unpredictable deployment state.
- Omitting dependency lockfiles (package-lock.json, go.sum, Cargo.lock) from continuous integration builds when configuring Incident Decision Framework Omitting lockfiles allows upstream package updates to break builds unpredictably without any changes to application codebase.
- Failing to define dynamic cache invalidation keys for CI package managers and layer caches when configuring Incident Decision Framework Stale build caches mask underlying dependency incompatibilities or pull outdated libraries, while un-cached runs slow feedback loops.