Staff SRE / Principal Architect [L3] Terraform Workspaces & CI/CD Staff SRE Scenario [L3]

Q: How do you run Terraform safely in a CI/CD pipeline? What are the guardrails?

\"When terraform plan shows unexpected changes, my golden rule is: never apply blindly. Investigate the diff first. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix.\" Key Strategy: Pro-Tip: Remote backend (S3 + DynamoDB locking) — never local state in CI..

State handling:

#Terraform #Workspaces & CI/CD #L3 #IaC #Cloud Infrastructure #S3

🎙️ Candidate Opening & Architectural Context

""When terraform plan shows unexpected changes, my golden rule is: never apply blindly. Investigate the diff first. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix.""

🛠️ Production Runbook & Step-by-Step Resolution

1️⃣

Initial Diagnostics & Root Cause Analysis

State handling:

Remote backend (S3 + DynamoDB locking) — never local state in CI.
Each pipeline run acquires lock before apply, releases after.
terraform plan -out=plan.tfplan in one stage.
Human reviews the plan (or automated check for unexpected destroys).
terraform apply plan.tfplan in a separate stage.
Fail the pipeline if plan shows any destroy without explicit override.
Run terraform fmt -check to fail on unformatted code.

2️⃣

Remediation & Permanent Safeguards

Plan before apply: Guardrails: No developer applies directly:

Run terraform validate to check syntax.
Run tflint for provider-specific lint rules.
Run tfsec or checkov for security misconfigurations.
Separate pipelines for different environments. Production requires manual approval.
All Terraform runs go through CI.
Developers open PRs → plan runs → review → merge → apply runs.

💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)

"Pro-Tip: Remote backend (S3 + DynamoDB locking) — never local state in CI.."

⚡ 60-Second Elevator Pitch Talking Points

Remote backend (S3 + DynamoDB locking) — never local state in CI.
Each pipeline run acquires lock before apply, releases after.
terraform plan -out=plan.tfplan in one stage.

Q: How do you run Terraform safely in a CI/CD pipeline? What are the guardrails?

🛠️ Production Runbook & Step-by-Step Resolution

Initial Diagnostics & Root Cause Analysis

Remediation & Permanent Safeguards

📚 Related Production Scenarios in Terraform