Q: A Terraform apply introduced a bad infrastructure change in production. What is the rollback process?
Revert the Terraform code to the last known good version and run a new plan to see what Terraform will change back.
Want to master this scenario in a live sandbox? KodeKloud's HashiCorp Certified Terraform Associate (003) Interactive Labs covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "Treat Terraform code with the same rigor as application code: pre-merge plans, state locks, and automated drift detection. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix."
Production Solution & Architecture
Revert the Terraform code to the last known good version and run a new plan to see what Terraform will change back. Apply that reviewed rollback plan through the normal approval path unless the incident process allows emergency approval. Restore state only if the state itself is wrong or corrupted; for a bad but successful infrastructure change, state usually reflects reality and the fix is another controlled apply. --- *More Terraform scenarios added periodically. PRs welcome.*
- Immediate Triage: Revert the Terraform code to the last known good version and run a new plan to see what Terrafo
- Run targeted verification commands before modifying configuration.
- Automate permanent guardrails (CI check, alerts, IaC policy) to prevent recurrence.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Executing Terraform applies without inspecting '# forces replacement' warnings in plan output when configuring Terraform apply introduced Attributes that force resource replacement destroy and recreate critical infrastructure (databases, load balancers), inducing unannounced production outages.
- Omitting remote state locking mechanisms (DynamoDB or backend native locks) for shared state files when configuring Terraform apply introduced Without state locking, concurrent CI/CD pipeline executions or team applies create race conditions that corrupt the state file.
- Structuring entire enterprise cloud footprints into a single monolithic Terraform root module when configuring Terraform apply introduced Monolithic state files slow plan execution dramatically, increase blast radius during failed applies, and create severe pipeline locking contention.
- Failing to pin required_providers versions, causing breaking provider upgrades in CI pipelines when configuring Terraform apply introduced Unpinned provider versions automatically download breaking major upgrades during CI runs, introducing unexpected schema validation failures.