Q: A critical production database in AWS RDS goes down due to an underlying hardware failure in `us-east-1a`. How does the application recover if you have Multi-AZ enabled? What is the expected downtime?
Multi-AZ RDS maintains a synchronous standby replica in a different Availability Zone (e.g., us-east-1b).
#General DevOps #General DevOps — Scenario-Based Interview Questions #L3 #DevOps #SRE #Architecture
🎙️ Candidate Opening & Architectural Context
""We faced this organizational and technical challenge while scaling our engineering teams. The interviewer is testing: RDS Multi-AZ failover mechanics, DNS TTL.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third.""
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Initial Diagnostics & Root Cause Analysis
Multi-AZ RDS maintains a synchronous standby replica in a different Availability Zone (e.g., us-east-1b).
- AWS detects the heartbeat failure.
- It automatically promotes the synchronous standby in
us-east-1bto become the new primary. - Crucially, AWS updates the backend CNAME DNS record of the database endpoint to point to the new primary's IP address.
2️⃣
Remediation & Permanent Safeguards
When the primary hardware fails: Downtime: The failover process typically takes 60 to 120 seconds. The application *will* experience database connection drops during this time. The application's database connection pool must be configured to automatically sever dead connections, resolve the DNS again (respecting the short TTL), and reconnect seamlessly to the new primary once it comes online.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: AWS detects the heartbeat failure.."
⚡ 60-Second Elevator Pitch Talking Points
- AWS detects the heartbeat failure.
- It automatically promotes the synchronous standby in us-east-1b to become the new primary.
- Crucially, AWS updates the backend CNAME DNS record of the database endpoint to point to the new ...
Advertisement