Q: How do you implement infrastructure drift detection?
To implement infrastructure drift detection, run an automated comparison between declared state (Terraform, CloudFormation) and real resources on a schedule, alert on any difference, and route each finding to a decision: revert.
Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: To implement <strong>infrastructure drift detection</strong>, run an automated comparison between declared state (Terraform, CloudFormation) and real resources on a schedule, alert on any difference, and route each finding to a decision: revert the change or codify it. Drift happens when someone changes a resource in the console or with a script, or when an external system mutates it. Interviewers ask this at senior level because detection is easy to demo but hard to operationalize: you need scheduling, noise control, ownership and prevention, so that drift becomes an exception rather than a monthly clean-up.
Define drift and choose detectors per layer
Different tools see different kinds of change.
- Terraform:
terraform planrefreshes state against the provider and shows differences from configuration. Use-refresh-onlyto see out-of-band changes without proposing config changes. - CloudFormation: built-in stack drift detection compares actual properties with the template for supported resource types; not every type is supported, so check the docs.
- AWS Config: records configuration changes over time and evaluates compliance rules, giving a timeline of who changed what and a drift-like signal for rule violations.
- Event-driven: CloudTrail plus EventBridge can alert in near real time on console or CLI write calls by humans.
- Layered answer: no single tool sees everything, so combine a scheduled IaC diff for declared resources with Config and CloudTrail for resources and actors outside the code.
- Scope first: list which accounts, environments and state files are in scope; unmanaged accounts cannot be checked by a plan job.
Hands-on: infrastructure drift detection with Terraform in CI
Schedule a plan and use its exit code as the signal.
name: drift-detection
on:
schedule: [{cron: "0 6 * * *"}]
workflow_dispatch: {}
permissions: {id-token: write, contents: read}
jobs:
plan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
- uses: aws-actions/configure-aws-credentials@v4
with: {role-to-assume: "arn:aws:iam::111122223333:role/tf-drift-readonly", aws-region: us-east-1}
- run: terraform init -input=false
- id: plan
run: |
set +e
terraform plan -refresh-only -detailed-exitcode -input=false -no-color | tee plan.txt
echo "code=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"
- if: steps.plan.outputs.code == '2'
run: echo "Drift detected" && exit 1 # replace with Slack/ticket notification
- Exit codes: with
-detailed-exitcode, 0 means no changes, 1 means error, 2 means changes present (drift or unapplied code). - Read-only credentials: the drift job needs only read access plus state read; do not give it apply rights.
- Separate real drift from pending code: run against the default branch so any diff means drift, not unmerged work.
- Ignore noise deliberately: use
lifecycle { ignore_changes = [...] }for attributes legitimately managed elsewhere, such as autoscaling desired capacity. - State access: the drift role needs read access to the remote state bucket and lock table; grant nothing broader.
- Matrix by workspace: use a job matrix over each root module or workspace so one noisy stack does not hide others.
Hands-on: CloudFormation drift detection
Start detection, poll it, then list differing resources.
ID=$(aws cloudformation detect-stack-drift --stack-name my-stack --query StackDriftDetectionId --output text)
aws cloudformation describe-stack-drift-detection-status --stack-drift-detection-id "$ID" \
--query '[DetectionStatus,StackDriftStatus]'
aws cloudformation describe-stack-resource-drifts --stack-name my-stack \
--stack-resource-drift-status-filters MODIFIED DELETED \
--query 'StackResourceDrifts[].[LogicalResourceId,StackResourceDriftStatus,PropertyDifferences]'
- Start: detection is asynchronous and returns a detection ID.
- Review: filter resource drifts to
MODIFIEDandDELETEDand read the property differences. - Schedule: trigger it from EventBridge Scheduler and a Lambda, or a CI job, and publish results to your alert channel.
- Limits: it covers properties the template defines; changes to properties not specified in the template may not be reported.
- Consolidate output: post a summary listing stack, resource logical IDs and the property differences so reviewers need not open the console.
Triage and remediation: revert or codify
Every finding needs an owner and a decision.
- Revert: if the manual change was wrong or an incident hot-fix that is over, run
terraform apply(or update the stack) to restore declared state. - Codify: if the change should stay, update the code to match; for resources created outside IaC use Terraform 1.5+
importblocks andterraform plan -generate-config-outas a starting point. - Accept and ignore: document attributes that are expected to differ and ignore them in code.
- Track: open a ticket or message in a channel with the plan summary, resource addresses and the CloudTrail principal who made the change.
- Hot-fix etiquette: during incidents, allow break-glass changes, but require a follow-up pull request within an agreed period so drift is reconciled, not forgotten.
Prevention and guardrails
Reduce drift at the source.
- Least privilege: give humans read-only production access; make changes through pipelines with a break-glass role that is audited.
- SCPs and Config rules: deny or flag high-risk manual changes such as opening security groups to the world.
- Real-time alerts: EventBridge rule on CloudTrail write events from non-pipeline principals.
- Measure: track drift count and time-to-resolve as team metrics.
- Avoid abandoned tools: prefer maintained mechanisms such as the scheduled plan above; community tools like driftctl are no longer actively developed.
- Tags for ownership: tag resources with a team and repo so alerts route to the right people automatically.
- Drift is when real infrastructure differs from the declared IaC state, usually from manual changes.
- I detect it with a scheduled terraform plan using detailed exit codes, CloudFormation drift detection and AWS Config, plus CloudTrail alerts.
- I triage each finding as revert or codify, use import blocks for adoption, and prevent drift with read-only access and SCPs.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Running drift checks from a feature branch Unmerged changes appear as differences and drown real drift in noise.
- Giving the drift job apply permissions A detection job should be read-only; auto-applying hides unreviewed changes and risks outages.
- Detecting drift without an owner or SLA Alerts get ignored and the backlog grows until nobody trusts the report.
- Assuming CloudFormation drift covers every resource and property Unsupported resource types and unspecified properties are not reported.