⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Linux & SRE Interview Questions Scenario 24 of 135 in Linux & SRE
Senior DevOps / SRE [L2] General DevOps General DevOps — Scenario-Based Interview Questions Production Scenario [L2]
🎯 Target Role / Context: General DevOps Scenario #20 (L2) · General DevOps — Scenario-Based Interview Questions

Q: In Terraform, you have accidentally deleted an RDS instance definition from your `main.tf` file and hit `apply`. Terraform is now attempting to destroy the production database. How do you architect Terraform files to protect against catastrophic accidental deletions?

While strict code reviews and IAM Deny policies are essential, Terraform has an explicit safety net for this exact scenario.

#General DevOps #L2 #DevOps #SRE #Architecture
🎙️ Candidate Opening & Architectural Context
""Senior DevOps is about building reliable automated feedback loops between code commit and production observability. The interviewer is testing: Terraform lifecycle rules, Prevent Destroy.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third.""
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? The Linux Foundation's Certified System Administrator (LFCS) & SRE Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: "Senior DevOps is about building reliable automated feedback loops between code commit and production observability. The interviewer is testing: Terraform lifecycle rules, Prevent Destroy.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."

1️⃣

Production Solution & Architecture

While strict code reviews and IAM Deny policies are essential, Terraform has an explicit safety net for this exact scenario. You should use the Terraform lifecycle block on any stateful, catastrophic-to-lose resources (like RDS, S3 buckets, DynamoDB tables). If a developer deletes this block or attempts to run terraform destroy, Terraform will parse the state file, observe the prevent_destroy metadata still attached to that resource, and flat-out refuse to execute the plan, throwing an error and saving the database.

resource "aws_db_instance" "production" {
  # ... configuration ...
  lifecycle {
    prevent_destroy = true
  }
}
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: While strict code reviews and IAM Deny policies are essential, Terraform has an explicit safety net for this exact scenario.."
⚡ 60-Second Elevator Pitch Talking Points
  • Immediate Triage: While strict code reviews and IAM Deny policies are essential, Terraform has an explicit safety
  • Run targeted verification commands before modifying configuration.
  • Automate permanent guardrails (CI check, alerts, IaC policy) to prevent recurrence.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Engineering bespoke in-house developer tools instead of adopting standardized platform Golden Paths when configuring General Terraform accidentally Bespoke internal tools create long-term maintenance debt and fragmented workflows, whereas standardized Golden Paths accelerate delivery.
  2. Overlooking infrastructure cost visibility and FinOps allocation tagging across multi-tenant environments when configuring General Terraform accidentally Without granular resource tagging, engineering organizations cannot attribute cloud infrastructure costs to specific teams or business units.
  3. Failing to define explicit Service Level Objectives (SLOs) and Error Budgets to balance feature velocity with reliability when configuring General Terraform accidentally Without agreed SLOs, engineering and product teams lack an objective data-driven framework to prioritize technical debt over new features.
  4. Neglecting automated continuous drift detection between declarative configuration and deployed cloud state when configuring General Terraform accidentally Unmanaged drift accumulates over time, causing future automated deployments to fail or overwrite emergency manual adjustments.
Advertisement
Want more Linux & SRE scenarios?
Explore our complete collection of scenario-based Linux & SRE interview runbooks.
Browse All Linux & SRE Questions →