Q: Two developers ran `terraform apply` at the same time on the same workspace. What happened and how do you prevent it?
This is a race condition. The last write wins — whichever apply finishes last overwrites the state file. This can cause state corruption ...
#Terraform #State & Locking #L2 #IaC #Cloud Infrastructure #S3
🎙️ Candidate Opening & Architectural Context
""When terraform plan shows unexpected changes, my golden rule is: never apply blindly. Investigate the diff first. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix.""
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Initial Diagnostics & Root Cause Analysis
This is a race condition. The last write wins — whichever apply finishes last overwrites the state file. This can cause state corruption and out-of-sync infrastructure.
- Use an S3 backend with DynamoDB locking: Terraform writes a lock entry to DynamoDB before applying. If another apply is running, the lock is already taken and the second apply waits or fails.
- Terraform Cloud/HCE automatically handles locking.
- Never use local state files for team work — they can't be locked.
2️⃣
Remediation & Permanent Safeguards
Prevention — state locking: Best practice: Run Terraform only from CI/CD pipelines, never from developer laptops directly. The pipeline enforces sequential execution.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Use an S3 backend with DynamoDB locking: Terraform writes a lock entry to DynamoDB before applying. If another apply is running, t."
⚡ 60-Second Elevator Pitch Talking Points
- Use an S3 backend with DynamoDB locking: Terraform writes a lock entry to DynamoDB before applyin...
- Terraform Cloud/HCE automatically handles locking.
- Never use local state files for team work — they can't be locked.
Advertisement