Q: Your company’s primary region (`us-east-1`) completely goes offline (a major AWS outage affecting the whole region). You are tasked with deciding when to trigger the Disaster Recovery failover to `us-west-2`. What metrics/business factors dictate this decision?
A regional failover is the most destructive action an SRE can take; it risks massive data loss, split-brain scenarios, and hours of complex resolution. It should not be triggered lightly.
Want to master this scenario in a live sandbox? The Linux Foundation's Certified System Administrator (LFCS) & SRE Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "We faced this organizational and technical challenge while scaling our engineering teams. The interviewer is testing: RTO, RPO, split-brain, cost of failover vs cost of downtime.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Initial Diagnostics & Root Cause Analysis
A regional failover is the most destructive action an SRE can take; it risks massive data loss, split-brain scenarios, and hours of complex resolution. It should not be triggered lightly.
- Is the outage estimated to last longer than our business-defined RTO? (e.g., if RTO is 4 hours, and AWS says it's an hour fix, we wait. If AWS says ETA unknown, we fail over).
- What is the state of the cross-region database replication? If the replication lag was 2 hours behind when the region died, failing over now guarantees 2 hours of permanent data loss (violating a strict 1-hour RPO).
- We must ensure the
us-east-1environment is completely fenced off (DNS isolated) to prevent a "split-brain" where systems come back online and start writing conflicting data to the old primary database while the newus-west-2primary is active.
Remediation & Permanent Safeguards
The decision is governed by RTO (Recovery Time Objective) and RPO (Recovery Point Objective):
- Is the outage estimated to last longer than our business-defined RTO? (e.g., if RTO is 4 hours, a...
- What is the state of the cross-region database replication? If the replication lag was 2 hours be...
- We must ensure the us-east-1 environment is completely fenced off (DNS isolated) to prevent a "sp...
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Engineering bespoke in-house developer tools instead of adopting standardized platform Golden Paths when configuring General company’s primary Bespoke internal tools create long-term maintenance debt and fragmented workflows, whereas standardized Golden Paths accelerate delivery.
- Overlooking infrastructure cost visibility and FinOps allocation tagging across multi-tenant environments when configuring General company’s primary Without granular resource tagging, engineering organizations cannot attribute cloud infrastructure costs to specific teams or business units.
- Failing to define explicit Service Level Objectives (SLOs) and Error Budgets to balance feature velocity with reliability when configuring General company’s primary Without agreed SLOs, engineering and product teams lack an objective data-driven framework to prioritize technical debt over new features.
- Neglecting automated continuous drift detection between declarative configuration and deployed cloud state when configuring General company’s primary Unmanaged drift accumulates over time, causing future automated deployments to fail or overwrite emergency manual adjustments.