Q: Someone manually changes Terraform-managed infrastructure — what happens?
Exhaustive breakdown of configuration drift in Terraform: what happens during refresh, how Terraform resolves divergence, and how to eliminate console access in production.
Terraform Drift: Someone Changed Production! What Happens?
Watch Naveed break down what happens when someone manually modifies Terraform-managed infrastructure—from state refresh and drift detection to remediation paths and production console safeguards in 60 seconds.
- State refresh queries cloud provider APIs to compare actual resources against terraform.tfstate
- Modified resources trigger in-place update proposals to revert back to code definition
- Deleted resources trigger proposed recreation from scratch to restore state
- Root governance: Zero human write access in AWS Production Console + daily automated drift CI scans
Want to master this scenario in a live sandbox? KodeKloud's HashiCorp Certified Terraform Associate (003) Interactive Labs covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: This is known as 'Configuration Drift'. What happens depends on whether the resource was modified, added, or deleted, and when the next Terraform execution runs.
What Happens During Next Terraform Run
During the next terraform plan or apply, Terraform performs a refresh:
- State Refresh: Terraform queries the cloud provider API (e.g. AWS API) for all resources tracked in
terraform.tfstate. - If a tracked resource was modified manually: Terraform compares the real-world state against the code. It flags drift and proposes an in-place update to revert the resource back to the code definition.
- If a tracked resource was deleted manually: Terraform state detects the missing resource and proposes recreating it from scratch.
- If unmanaged sub-resources were added manually: For example, someone manually added tags or security group rules: Depending on resource arguments, Terraform may delete the extra rules, ignore them, or fail with conflict errors.
Two Remediation Paths
The team must choose between restoring code or adopting the change:
- Path A (Enforce Code / Revert Drift): Run
terraform apply. Terraform will overwrite the manual console changes and restore the infrastructure to the version-controlled state of truth. - Path B (Adopt the Manual Change into Code): If the manual change was an emergency hotfix: update the
.tfcode to match the new configuration, runterraform planto verify 0 proposed changes, and commit the code.
How to Prevent It from Happening Again
Enterprise governance safeguards:
- Remove Write Access in Production: Developers and DevOps engineers should have
ReadOnlyAccessin the AWS Production Console. Only the CI/CD pipeline's IAM role should have write permissions. - Automated Drift Detection: Run a scheduled daily CI/CD pipeline (e.g. at 2 AM) executing
terraform plan -detailed-exitcode. If exit code is 2 (drift detected), send an automated alert to Slack/PagerDuty. - Use SCPs (Service Control Policies): Enforce AWS Organizations SCPs preventing direct edits to mission-critical VPC, IAM, or KMS resources.
- What happens: On next 'terraform plan', refresh detects divergence between cloud reality and state. Terraform proposes reverting manual edits to match code.
- If resource deleted manually: Terraform proposes recreating it.
- Remediation: Either 'terraform apply' to overwrite manual changes, or update code to match reality and commit.
- Root prevention: Restrict IAM - zero human write access in Production AWS Console; only CI/CD IAM role has write access.
- Detection: Schedule daily automated 'terraform plan -detailed-exitcode' in CI; alert team on exit code 2 (drift).
Mastered this procedure? Subscribe to @naveedkumbhar01 on YouTube for daily 60-second SRE incident runbooks, Kubernetes triage shortcuts, and high-impact interview elevator pitches.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Executing Terraform applies without inspecting '# forces replacement' warnings in plan output when configuring Someone Manually Changes Attributes that force resource replacement destroy and recreate critical infrastructure (databases, load balancers), inducing unannounced production outages.
- Omitting remote state locking mechanisms (DynamoDB or backend native locks) for shared state files when configuring Someone Manually Changes Without state locking, concurrent CI/CD pipeline executions or team applies create race conditions that corrupt the state file.
- Structuring entire enterprise cloud footprints into a single monolithic Terraform root module when configuring Someone Manually Changes Monolithic state files slow plan execution dramatically, increase blast radius during failed applies, and create severe pipeline locking contention.
- Failing to pin required_providers versions, causing breaking provider upgrades in CI pipelines when configuring Someone Manually Changes Unpinned provider versions automatically download breaking major upgrades during CI runs, introducing unexpected schema validation failures.