Q: Your team uses Terraform. Developer A runs `terraform apply`, but Developer B runs `terraform apply` on the same directory at the exact same time. What happens, and what mechanism should be in place to prevent disaster?
If they are using local state or a remote state backend without locking (e.g., basic S3 only), both executions will run simultaneously. T...
🛠️ Production Runbook & Step-by-Step Resolution
Production Solution & Architecture
If they are using local state or a remote state backend without locking (e.g., basic S3 only), both executions will run simultaneously. They will race to update the cloud APIs, resulting in massive resource conflict, split-brain infrastructure, and guaranteed corruption of the terraform.tfstate file. To prevent this, production DevOps teams use a Remote Backend with State Locking. The standard AWS architecture is storing the state file in S3, and using a DynamoDB Table for the lock. When Developer A runs apply, Terraform automatically writes a lock entry to DynamoDB. When Developer B runs apply a millisecond later, Terraform checks DynamoDB, sees the lock, and immediately aborts Developer B's run with a Lock exists error, safely protecting the state.
- Immediate Triage: If they are using local state or a remote state backend without locking (e.g., basic S3 only),
- Run targeted verification commands before modifying configuration.
- Automate permanent guardrails (CI check, alerts, IaC policy) to prevent recurrence.