Q: How do you handle cross-region disaster recovery with Terraform?
Cross-region disaster recovery with Terraform means describing both the primary and the recovery region as code, using provider aliases and reusable modules, while the actual data replication is handled by the services themselves.
Want to master this scenario in a live sandbox? KodeKloud's HashiCorp Certified Terraform Associate (003) Interactive Labs covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Cross-region disaster recovery with Terraform means describing both the primary and the recovery region as code, using provider aliases and reusable modules, while the actual data replication is handled by the services themselves (S3 replication, Aurora Global Database, DynamoDB global tables). Terraform builds and keeps the DR footprint consistent; it does not fail traffic over by itself. Interviewers ask this to see whether you separate state per region, avoid making the DR region depend on the failed region, pick a pilot light or warm standby model deliberately, and rehearse the failover rather than just writing the code.
Decide the DR model and isolate state
The recovery tier (backup and restore, pilot light, warm standby, active-active) drives cost, RTO and RPO, and therefore the Terraform layout.
- Pilot light: data stores replicate continuously, compute exists at zero or minimal capacity, and you scale it up during failover. Cheap, but RTO includes scale-up time.
- Warm standby: a scaled-down but running copy of the stack. Faster failover, higher steady-state cost.
- State layout: keep a separate state per region (for example
prod/us-east-1andprod/us-west-2), and keep the DR state bucket outside the primary region so you can still runapplywhen the primary is down. - Shared code, different inputs: one module, two tfvars files, so the DR stack cannot drift from production.
- RTO and RPO first: write down the recovery time and recovery point objectives per tier before choosing a model, because they decide whether continuous replication or periodic backups are acceptable.
Hands-on: cross-region disaster recovery with Terraform
Use provider aliases and pass each one into the same module.
provider "aws" {
region = "us-east-1"
}
provider "aws" {
alias = "dr"
region = "us-west-2"
}
module "app_primary" {
source = "./modules/app"
providers = { aws = aws }
desired_capacity = 6
}
module "app_dr" {
source = "./modules/app"
providers = { aws = aws.dr }
desired_capacity = 1 # pilot light; raise during failover
}
- Provider alias: declare a second
awsprovider withalias = "dr"and a different region. - Module providers map: use
providers = { aws = aws.dr }so every resource in the module lands in the DR region. - Capacity variable: expose
desired_capacityso DR runs at reduced size and can be raised during failover. - Global resources: IAM and Route 53 are global; create them once, not per region, to avoid duplicates and conflicts.
Replicate data and wire traffic failover
Terraform declares the replication and routing resources; the services move the data.
resource "aws_route53_record" "primary" {
zone_id = var.zone_id
name = "app.example.com"
type = "CNAME"
ttl = 60
set_identifier = "primary"
records = [module.app_primary.lb_dns]
health_check_id = aws_route53_health_check.primary.id
failover_routing_policy { type = "PRIMARY" }
}
- Object storage: enable versioning on both buckets and add
aws_s3_bucket_replication_configurationwith an IAM role that can replicate. - Databases: use Aurora Global Database or DynamoDB global tables where supported; check current regional support and promotion behavior for your engine.
- Encryption: use multi-Region KMS keys or a destination-region key for each replicated store so replicas can be decrypted after failover.
- DNS: create Route 53 failover records tied to health checks so traffic moves to the secondary endpoint when the primary is unhealthy.
- Cross-region copies: use cross-region snapshot copies and AMI copies for tiers that tolerate a longer recovery time, which costs far less than a running standby.
Failure modes to design out
Most DR plans fail on hidden dependencies, not on the Terraform syntax.
- Hidden regional dependencies: ECR images, secrets, AMIs, parameter store values and certificates must exist in the DR region. Copy or replicate them explicitly.
- Split-brain on promotion: promoting a replica while the old primary is still writing risks divergence; define who can trigger failover and how the old primary is fenced.
- Quota and capacity: check current service quotas in the DR region, and consider capacity reservations for critical instance types.
- Drift: run a scheduled
terraform plan -detailed-exitcodeagainst the DR stack so manual changes are detected. - Config drift between regions: do not hand-tune the DR region. Every manual change should be backported into the shared module, otherwise failover day is the first time the stacks differ.
Verify with drills and automate
An untested DR stack is a hope, not a plan.
- Game days: periodically fail traffic over in a non-production copy or a controlled window, measure the real RTO and RPO and record them.
- Pipeline: apply primary and DR from the same pipeline with the same module version, gated by plan review.
- Failback: document and rehearse returning to the primary, including data resynchronisation.
- Runbook as code: keep the failover steps (promote database, switch DNS weight, raise desired capacity) in a scripted, reviewed procedure with a clear approval owner.
- Observability: monitor replication lag and health checks from outside the primary region so the signal that triggers failover survives the outage.
- Cross-region DR in Terraform is one module deployed twice through provider aliases, with per-region state and service-level replication.
- I verify with scheduled plans for drift and regular failover drills that measure actual RTO and RPO.
- I automate both regions from one pipeline and keep state and secrets available outside the primary region.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Storing the only state bucket in the primary region A regional outage blocks the Terraform run you need to recover.
- Assuming Terraform replicates data It only creates replication resources; promotion and consistency are service behaviors you must understand and test.
- Forgetting regional copies of images, keys and secrets The DR stack boots but cannot pull images or decrypt data.
- Never rehearsing failover Hidden dependencies and quota limits only appear under a real switchover.