⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All AWS & Cloud Architecture Interview Questions Scenario 89 of 179 in AWS & Cloud Architecture
Staff SRE / Principal Architect [L3] AWS Cost & Architecture Staff SRE Scenario [L3]
🎯 Target Role / Context: AWS Scenario #89 (L3) · Cost & Architecture

Q: How do you implement infrastructure drift detection?

To implement infrastructure drift detection, run an automated comparison between declared state (Terraform, CloudFormation) and real resources on a schedule, alert on any difference, and route each finding to a decision: revert.

#AWS #Cost & Architecture #L3 #Cloud #Infrastructure #Terraform State
🎙️ Candidate Opening & Architectural Context
"To implement <strong>infrastructure drift detection</strong>, run an automated comparison between declared state (Terraform, CloudFormation) and real resources on a schedule, alert on any difference, and route each finding to a decision: revert the change or codify it. Drift happens when someone changes a resource in the console or with a script, or when an external system mutates it. Interviewers ask this at senior level because detection is easy to demo but hard to operationalize: you need scheduling, noise control, ownership and prevention, so that drift becomes an exception rather than a monthly clean-up."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: To implement <strong>infrastructure drift detection</strong>, run an automated comparison between declared state (Terraform, CloudFormation) and real resources on a schedule, alert on any difference, and route each finding to a decision: revert the change or codify it. Drift happens when someone changes a resource in the console or with a script, or when an external system mutates it. Interviewers ask this at senior level because detection is easy to demo but hard to operationalize: you need scheduling, noise control, ownership and prevention, so that drift becomes an exception rather than a monthly clean-up.

1️⃣

Define drift and choose detectors per layer

Different tools see different kinds of change.

  • Terraform: terraform plan refreshes state against the provider and shows differences from configuration. Use -refresh-only to see out-of-band changes without proposing config changes.
  • CloudFormation: built-in stack drift detection compares actual properties with the template for supported resource types; not every type is supported, so check the docs.
  • AWS Config: records configuration changes over time and evaluates compliance rules, giving a timeline of who changed what and a drift-like signal for rule violations.
  • Event-driven: CloudTrail plus EventBridge can alert in near real time on console or CLI write calls by humans.
  • Layered answer: no single tool sees everything, so combine a scheduled IaC diff for declared resources with Config and CloudTrail for resources and actors outside the code.
  • Scope first: list which accounts, environments and state files are in scope; unmanaged accounts cannot be checked by a plan job.
2️⃣

Hands-on: infrastructure drift detection with Terraform in CI

Schedule a plan and use its exit code as the signal.

name: drift-detection
on:
  schedule: [{cron: "0 6 * * *"}]
  workflow_dispatch: {}
permissions: {id-token: write, contents: read}
jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - uses: aws-actions/configure-aws-credentials@v4
        with: {role-to-assume: "arn:aws:iam::111122223333:role/tf-drift-readonly", aws-region: us-east-1}
      - run: terraform init -input=false
      - id: plan
        run: |
          set +e
          terraform plan -refresh-only -detailed-exitcode -input=false -no-color | tee plan.txt
          echo "code=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"
      - if: steps.plan.outputs.code == '2'
        run: echo "Drift detected" && exit 1   # replace with Slack/ticket notification
  • Exit codes: with -detailed-exitcode, 0 means no changes, 1 means error, 2 means changes present (drift or unapplied code).
  • Read-only credentials: the drift job needs only read access plus state read; do not give it apply rights.
  • Separate real drift from pending code: run against the default branch so any diff means drift, not unmerged work.
  • Ignore noise deliberately: use lifecycle { ignore_changes = [...] } for attributes legitimately managed elsewhere, such as autoscaling desired capacity.
  • State access: the drift role needs read access to the remote state bucket and lock table; grant nothing broader.
  • Matrix by workspace: use a job matrix over each root module or workspace so one noisy stack does not hide others.
Advertisement
3️⃣

Hands-on: CloudFormation drift detection

Start detection, poll it, then list differing resources.

ID=$(aws cloudformation detect-stack-drift --stack-name my-stack --query StackDriftDetectionId --output text)
aws cloudformation describe-stack-drift-detection-status --stack-drift-detection-id "$ID" \
  --query '[DetectionStatus,StackDriftStatus]'

aws cloudformation describe-stack-resource-drifts --stack-name my-stack \
  --stack-resource-drift-status-filters MODIFIED DELETED \
  --query 'StackResourceDrifts[].[LogicalResourceId,StackResourceDriftStatus,PropertyDifferences]'
  • Start: detection is asynchronous and returns a detection ID.
  • Review: filter resource drifts to MODIFIED and DELETED and read the property differences.
  • Schedule: trigger it from EventBridge Scheduler and a Lambda, or a CI job, and publish results to your alert channel.
  • Limits: it covers properties the template defines; changes to properties not specified in the template may not be reported.
  • Consolidate output: post a summary listing stack, resource logical IDs and the property differences so reviewers need not open the console.
4️⃣

Triage and remediation: revert or codify

Every finding needs an owner and a decision.

  • Revert: if the manual change was wrong or an incident hot-fix that is over, run terraform apply (or update the stack) to restore declared state.
  • Codify: if the change should stay, update the code to match; for resources created outside IaC use Terraform 1.5+ import blocks and terraform plan -generate-config-out as a starting point.
  • Accept and ignore: document attributes that are expected to differ and ignore them in code.
  • Track: open a ticket or message in a channel with the plan summary, resource addresses and the CloudTrail principal who made the change.
  • Hot-fix etiquette: during incidents, allow break-glass changes, but require a follow-up pull request within an agreed period so drift is reconciled, not forgotten.
5️⃣

Prevention and guardrails

Reduce drift at the source.

  • Least privilege: give humans read-only production access; make changes through pipelines with a break-glass role that is audited.
  • SCPs and Config rules: deny or flag high-risk manual changes such as opening security groups to the world.
  • Real-time alerts: EventBridge rule on CloudTrail write events from non-pipeline principals.
  • Measure: track drift count and time-to-resolve as team metrics.
  • Avoid abandoned tools: prefer maintained mechanisms such as the scheduled plan above; community tools like driftctl are no longer actively developed.
  • Tags for ownership: tag resources with a team and repo so alerts route to the right people automatically.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Run a daily read-only terraform plan -refresh-only -detailed-exitcode on the default branch and treat exit code 2 as an alert with a named owner."
⚡ 60-Second Elevator Pitch Talking Points
  • Drift is when real infrastructure differs from the declared IaC state, usually from manual changes.
  • I detect it with a scheduled terraform plan using detailed exit codes, CloudFormation drift detection and AWS Config, plus CloudTrail alerts.
  • I triage each finding as revert or codify, use import blocks for adoption, and prevent drift with read-only access and SCPs.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Running drift checks from a feature branch Unmerged changes appear as differences and drown real drift in noise.
  2. Giving the drift job apply permissions A detection job should be read-only; auto-applying hides unreviewed changes and risks outages.
  3. Detecting drift without an owner or SLA Alerts get ignored and the backlog grows until nobody trusts the report.
  4. Assuming CloudFormation drift covers every resource and property Unsupported resource types and unspecified properties are not reported.
Advertisement
Want more AWS & Cloud Architecture scenarios?
Explore our complete collection of scenario-based AWS & Cloud Architecture interview runbooks.
Browse All AWS & Cloud Architecture Questions →