⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Terraform & IaC Interview Questions Scenario 37 of 120 in Terraform & IaC
Staff SRE / Principal Architect [L3] Terraform Terraform Modules & State Staff SRE Scenario [L3]
🎯 Target Role / Context: Terraform Scenario #32 (L3) · Use VPC ID from another module

Q: How do you handle cross-region disaster recovery with Terraform?

Cross-region disaster recovery with Terraform means describing both the primary and the recovery region as code, using provider aliases and reusable modules, while the actual data replication is handled by the services themselves.

#Terraform #L3 #IaC #Cloud Infrastructure
🎙️ Candidate Opening & Architectural Context
"Cross-region disaster recovery with Terraform means describing both the primary and the recovery region as code, using provider aliases and reusable modules, while the actual data replication is handled by the services themselves (S3 replication, Aurora Global Database, DynamoDB global tables). Terraform builds and keeps the DR footprint consistent; it does not fail traffic over by itself. Interviewers ask this to see whether you separate state per region, avoid making the DR region depend on the failed region, pick a pilot light or warm standby model deliberately, and rehearse the failover rather than just writing the code."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's HashiCorp Certified Terraform Associate (003) Interactive Labs covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: Cross-region disaster recovery with Terraform means describing both the primary and the recovery region as code, using provider aliases and reusable modules, while the actual data replication is handled by the services themselves (S3 replication, Aurora Global Database, DynamoDB global tables). Terraform builds and keeps the DR footprint consistent; it does not fail traffic over by itself. Interviewers ask this to see whether you separate state per region, avoid making the DR region depend on the failed region, pick a pilot light or warm standby model deliberately, and rehearse the failover rather than just writing the code.

1️⃣

Decide the DR model and isolate state

The recovery tier (backup and restore, pilot light, warm standby, active-active) drives cost, RTO and RPO, and therefore the Terraform layout.

  • Pilot light: data stores replicate continuously, compute exists at zero or minimal capacity, and you scale it up during failover. Cheap, but RTO includes scale-up time.
  • Warm standby: a scaled-down but running copy of the stack. Faster failover, higher steady-state cost.
  • State layout: keep a separate state per region (for example prod/us-east-1 and prod/us-west-2), and keep the DR state bucket outside the primary region so you can still run apply when the primary is down.
  • Shared code, different inputs: one module, two tfvars files, so the DR stack cannot drift from production.
  • RTO and RPO first: write down the recovery time and recovery point objectives per tier before choosing a model, because they decide whether continuous replication or periodic backups are acceptable.
Pro Tip: If your only state bucket and lock mechanism live in the primary region, a regional outage also blocks the Terraform run you need to recover.
2️⃣

Hands-on: cross-region disaster recovery with Terraform

Use provider aliases and pass each one into the same module.

provider "aws" {
  region = "us-east-1"
}

provider "aws" {
  alias  = "dr"
  region = "us-west-2"
}

module "app_primary" {
  source           = "./modules/app"
  providers        = { aws = aws }
  desired_capacity = 6
}

module "app_dr" {
  source           = "./modules/app"
  providers        = { aws = aws.dr }
  desired_capacity = 1   # pilot light; raise during failover
}
  • Provider alias: declare a second aws provider with alias = "dr" and a different region.
  • Module providers map: use providers = { aws = aws.dr } so every resource in the module lands in the DR region.
  • Capacity variable: expose desired_capacity so DR runs at reduced size and can be raised during failover.
  • Global resources: IAM and Route 53 are global; create them once, not per region, to avoid duplicates and conflicts.
Advertisement
3️⃣

Replicate data and wire traffic failover

Terraform declares the replication and routing resources; the services move the data.

resource "aws_route53_record" "primary" {
  zone_id         = var.zone_id
  name            = "app.example.com"
  type            = "CNAME"
  ttl             = 60
  set_identifier  = "primary"
  records         = [module.app_primary.lb_dns]
  health_check_id = aws_route53_health_check.primary.id
  failover_routing_policy { type = "PRIMARY" }
}
  • Object storage: enable versioning on both buckets and add aws_s3_bucket_replication_configuration with an IAM role that can replicate.
  • Databases: use Aurora Global Database or DynamoDB global tables where supported; check current regional support and promotion behavior for your engine.
  • Encryption: use multi-Region KMS keys or a destination-region key for each replicated store so replicas can be decrypted after failover.
  • DNS: create Route 53 failover records tied to health checks so traffic moves to the secondary endpoint when the primary is unhealthy.
  • Cross-region copies: use cross-region snapshot copies and AMI copies for tiers that tolerate a longer recovery time, which costs far less than a running standby.
4️⃣

Failure modes to design out

Most DR plans fail on hidden dependencies, not on the Terraform syntax.

  • Hidden regional dependencies: ECR images, secrets, AMIs, parameter store values and certificates must exist in the DR region. Copy or replicate them explicitly.
  • Split-brain on promotion: promoting a replica while the old primary is still writing risks divergence; define who can trigger failover and how the old primary is fenced.
  • Quota and capacity: check current service quotas in the DR region, and consider capacity reservations for critical instance types.
  • Drift: run a scheduled terraform plan -detailed-exitcode against the DR stack so manual changes are detected.
  • Config drift between regions: do not hand-tune the DR region. Every manual change should be backported into the shared module, otherwise failover day is the first time the stacks differ.
5️⃣

Verify with drills and automate

An untested DR stack is a hope, not a plan.

  • Game days: periodically fail traffic over in a non-production copy or a controlled window, measure the real RTO and RPO and record them.
  • Pipeline: apply primary and DR from the same pipeline with the same module version, gated by plan review.
  • Failback: document and rehearse returning to the primary, including data resynchronisation.
  • Runbook as code: keep the failover steps (promote database, switch DNS weight, raise desired capacity) in a scripted, reviewed procedure with a clear approval owner.
  • Observability: monitor replication lag and health checks from outside the primary region so the signal that triggers failover survives the outage.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Keep DR state and its locking outside the primary region, and treat the failover runbook (promotion, DNS, scale-up) as tested code, not a diagram."
⚡ 60-Second Elevator Pitch Talking Points
  • Cross-region DR in Terraform is one module deployed twice through provider aliases, with per-region state and service-level replication.
  • I verify with scheduled plans for drift and regular failover drills that measure actual RTO and RPO.
  • I automate both regions from one pipeline and keep state and secrets available outside the primary region.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Storing the only state bucket in the primary region A regional outage blocks the Terraform run you need to recover.
  2. Assuming Terraform replicates data It only creates replication resources; promotion and consistency are service behaviors you must understand and test.
  3. Forgetting regional copies of images, keys and secrets The DR stack boots but cannot pull images or decrypt data.
  4. Never rehearsing failover Hidden dependencies and quota limits only appear under a real switchover.
Advertisement
Want more Terraform & IaC scenarios?
Explore our complete collection of scenario-based Terraform & IaC interview runbooks.
Browse All Terraform & IaC Questions →