Q: In Terraform, you have accidentally deleted an RDS instance definition from your `main.tf` file and hit `apply`. Terraform is now attempting to destroy the production database. How do you architect Terraform files to protect against catastrophic accidental deletions?
While strict code reviews and IAM Deny policies are essential, Terraform has an explicit safety net for this exact scenario.
#General DevOps #General DevOps — Scenario-Based Interview Questions #L2 #DevOps #SRE #Architecture
🎙️ Candidate Opening & Architectural Context
""Senior DevOps is about building reliable automated feedback loops between code commit and production observability. The interviewer is testing: Terraform lifecycle rules, Prevent Destroy.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third.""
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Production Solution & Architecture
While strict code reviews and IAM Deny policies are essential, Terraform has an explicit safety net for this exact scenario. You should use the Terraform lifecycle block on any stateful, catastrophic-to-lose resources (like RDS, S3 buckets, DynamoDB tables). If a developer deletes this block or attempts to run terraform destroy, Terraform will parse the state file, observe the prevent_destroy metadata still attached to that resource, and flat-out refuse to execute the plan, throwing an error and saving the database.
resource "aws_db_instance" "production" {
# ... configuration ...
lifecycle {
prevent_destroy = true
}
}
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: While strict code reviews and IAM Deny policies are essential, Terraform has an explicit safety net for this exact scenario.."
⚡ 60-Second Elevator Pitch Talking Points
- Immediate Triage: While strict code reviews and IAM Deny policies are essential, Terraform has an explicit safety
- Run targeted verification commands before modifying configuration.
- Automate permanent guardrails (CI check, alerts, IaC policy) to prevent recurrence.
Advertisement