Q: How do you run Terraform safely in a CI/CD pipeline? What are the guardrails?
State handling:
Want to master this scenario in a live sandbox? KodeKloud's HashiCorp Certified Terraform Associate (003) Interactive Labs covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "When terraform plan shows unexpected changes, my golden rule is: never apply blindly. Investigate the diff first. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix."
Initial Diagnostics & Root Cause Analysis
State handling:
- Remote backend (S3 + DynamoDB locking) — never local state in CI.
- Each pipeline run acquires lock before apply, releases after.
terraform plan -out=plan.tfplanin one stage.- Human reviews the plan (or automated check for unexpected destroys).
terraform apply plan.tfplanin a separate stage.- Fail the pipeline if plan shows any
destroywithout explicit override. - Run
terraform fmt -checkto fail on unformatted code.
Remediation & Permanent Safeguards
Plan before apply: Guardrails: No developer applies directly:
- Run
terraform validateto check syntax. - Run
tflintfor provider-specific lint rules. - Run
tfsecorcheckovfor security misconfigurations. - Separate pipelines for different environments. Production requires manual approval.
- All Terraform runs go through CI.
- Developers open PRs → plan runs → review → merge → apply runs.
- Remote backend (S3 + DynamoDB locking) — never local state in CI.
- Each pipeline run acquires lock before apply, releases after.
- terraform plan -out=plan.tfplan in one stage.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Relying on legacy bucket ACLs instead of modern S3 Block Public Access and IAM bucket policies when configuring Terraform safely CI/CD AWS recommends disabling ACLs (Bucket Owner Enforced) in favor of centralized declarative bucket policies and account-level Block Public Access controls.
-
Confusing SSE-S3 encryption with KMS customer-managed key (SSE-KMS) authorization policies when configuring Terraform safely CI/CD
Accessing SSE-KMS encrypted objects requires both
s3:GetObjectandkms:Decryptpermissions; neglecting KMS key policies causes silent 403 Forbidden errors. - Using sequential timestamps or predictable prefixes for high-throughput S3 read and write keys when configuring Terraform safely CI/CD Sequential key prefixes limit S3 horizontal partitioning; randomized hash prefixes distribute workloads across partitions to achieve 3,500 PUT and 5,500 GET requests/sec.
- Failing to audit S3 object delete markers when troubleshooting missing versioned objects when configuring Terraform safely CI/CD Issuing a DELETE on a versioned bucket only adds a delete marker; engineers must retrieve previous versions or purge markers to restore data.