Q: An RDS instance went down and the automated failover to the standby didn't happen as expected in a Multi-AZ setup. What could have gone wrong?
Multi-AZ failover is automatic, but several things can prevent or delay it:
Want to master this scenario in a live sandbox? KodeKloud's PostgreSQL Database Administration & High Availability Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "In our AWS cloud environment, we managed high-traffic microservices where this exact scenario occurred. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix."
Initial Diagnostics & Root Cause Analysis
Multi-AZ failover is automatic, but several things can prevent or delay it:
- Maintenance mode — if you disabled automatic failover before maintenance.
- Failover trigger conditions — RDS only fails over for: storage failure, loss of network connectivity, instance hardware failure, running out of storage. It does NOT auto-failover for high CPU/load.
- DNS propagation delay — failover promotes the standby, but the endpoint DNS (e.g.,
mydb.xxxxxx.us-east-1.rds.amazonaws.com) needs to update. If the app uses hardcoded IPs instead of the RDS DNS endpoint, failover doesn't help. Always use DNS endpoint.
Remediation & Permanent Safeguards
- Application doesn't retry connections — after failover, existing connections are dropped. The app must retry DB connections. Apps that don't retry see prolonged outage.
- Replication lag — in some edge cases, the standby might have a small lag, causing momentary data inconsistency.
- Maintenance mode — if you disabled automatic failover before maintenance.
- Failover trigger conditions — RDS only fails over for: storage failure, loss of network connectiv...
- DNS propagation delay — failover promotes the standby, but the endpoint DNS (e.g., mydb.xxxxxx.us...
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Applying exclusive-lock DDL table migrations on live production databases during peak traffic windows when configuring instance went automated DDL migrations that acquire an exclusive table lock block all concurrent queries, queuing connections until the database connection pool exhausts.
- Allowing high-frequency queries to trigger full table scans without appropriate composite indexing when configuring instance went automated Sequential table scans saturate database CPU and disk I/O, evicting frequently queried data from the buffer cache and causing cascading timeouts.
- Disabling referential integrity constraints during bulk ETL imports without post-import consistency validation when configuring instance went automated Skipping relational consistency checks risks introducing orphaned records and referential integrity violations into production tables.
- Relying on database snapshot backups that have never been verified through automated restoration drills when configuring instance went automated Backups that have never been tested with automated restore verification frequently fail during real disaster recovery scenarios.