Senior DevOps / SRE Terraform State & Collaboration Architecture & Best Practices

Q: How would you manage state for multiple engineers?

Production-grade remote backend architecture for team collaboration: S3 state storage, DynamoDB state locking, encryption, state isolation, and Atlantis/CI execution.

#Terraform #S3 #DynamoDB #State Locking #Remote Backend #Atlantis

🎙️ Candidate Opening & Architectural Context

"Managing Terraform state across a team requires eliminating local state files completely. We use a secure remote backend with distributed locking, encryption, versioning, and execution isolation."

🛠️ Production Runbook & Step-by-Step Resolution

1️⃣

AWS S3 + DynamoDB Remote Backend

The industry standard remote backend configuration:

Amazon S3 Bucket: Stores the terraform.tfstate file centrally.
S3 Bucket Versioning: MUST be enabled. If an engineer or corrupt apply corrupts state, you can roll back to previous state versions instantly.
Encryption at Rest: Enforce AES256 or aws:kms server-side encryption.
DynamoDB State Locking: Table with Primary Key LockID (String). When an engineer runs plan/apply, Terraform acquires a lock. If another engineer tries to apply simultaneously, Terraform outputs: Error: Error acquiring the state lock, preventing race conditions and state corruption.

2️⃣

State Decomposition & Blast Radius Reduction

Never put all infrastructure into a single monolithic state file:

Split state by Layer: networking/ (VPC), compute/ (EKS), data/ (RDS), security/ (IAM).
Split state by Environment: Dev, Staging, and Prod must NEVER share a state file.
Benefits: Smaller state files execute 10x faster, team members don't lock each other out, and a mistake in a compute resource cannot destroy the VPC.

3️⃣

Execute Through CI/CD (Atlantis / Terraform Cloud)

Engineers should not run 'apply' from local laptops:

Use Atlantis, Terraform Cloud, or GitHub Actions.
Engineers open a Pull Request. Atlantis automatically runs terraform plan and comments the plan diff directly on the PR.
Once reviewed and approved by senior engineers, someone types atlantis apply in PR comments. The execution happens strictly in CI with an audit log.

4️⃣

State Security & Sensitive Data

Protecting secrets inside state:

Terraform state contains plaintext sensitive values (e.g. generated DB passwords).
Lock down S3 bucket with strict bucket policy allowing access ONLY to the CI/CD execution role.
Block all public S3 bucket access.

💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)

"Use S3 with versioning and KMS encryption for state storage, DynamoDB for distributed locking, split state by layer and environment, and execute applies exclusively through CI/CD (Atlantis/PR workflow)."

⚡ 60-Second Elevator Pitch Talking Points

Remote Backend: S3 bucket with versioning (enables rollback of corrupt state) and KMS encryption.
State Locking: DynamoDB table with LockID key to prevent concurrent apply race conditions.
Decomposition: Split state by layer (network, compute, database) and environment (dev, staging, prod) to limit blast radius.
Access Control: S3 bucket policy allowing only CI/CD IAM role; block public access (state stores plaintext secrets).
Team Workflow: Run plan/apply via Atlantis or CI/CD PR workflow rather than individual local laptops.