Q: During a transient network partition between AWS Availability Zones, a Redis master node is isolated from its Sentinel monitors and replica instances. The Sentinels detect quorum loss, declare the master dead, and promote a replica in AZ-b to become the new master. However, client applications in AZ-a still have network access to the old partitioned master and continue executing writes for 90 seconds. When the network heals, all writes made to the old master are irrevocably lost. How do you configure Redis min-replicas-to-write and quorum failover semantics to prevent split-brain write loss?
Prevent silent data loss in Redis Sentinel clusters during network partitions by tuning quorum configurations and enforcing min-replicas-to-write rules.
Want to master this scenario in a live sandbox? KodeKloud's PostgreSQL Database Administration & High Availability Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Redis replication is asynchronous by default. When a network partition divides a Redis cluster, the Sentinel quorum promotes a replica to master in the majority partition. However, if clients in the minority partition continue writing to the partitioned old master, those writes cannot be synchronized. When the partition heals and the old master demotes to a replica, it resynchronizes from the new master by discarding its own local dataset, permanently deleting all unacknowledged minority writes.
Situation: Asynchronous Failover Causes 90 Seconds of Lost Writes
Task: Enforce Strict Write Quorum Guardrails on Master Instances
Action: Configure min-replicas-to-write & Sentinel Down-After-Milliseconds
Result: 0 Data Loss During Partition Drills
min-replicas-to-write 1 and min-replicas-max-lag so an isolated master stops accepting writes before Sentinels promote a new master, preventing split-brain data loss."
- Network partitions can cause split-brain where clients write to an isolated master that gets wiped upon healing.
- Configure min-replicas-to-write 1 and min-replicas-max-lag to reject writes on partitioned masters.
- Keep down-after-milliseconds low and distribute Sentinel instances across at least 3 availability zones.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Applying exclusive-lock DDL table migrations on live production databases during peak traffic windows when configuring Redis Sentinel Split DDL migrations that acquire an exclusive table lock block all concurrent queries, queuing connections until the database connection pool exhausts.
- Allowing high-frequency queries to trigger full table scans without appropriate composite indexing when configuring Redis Sentinel Split Sequential table scans saturate database CPU and disk I/O, evicting frequently queried data from the buffer cache and causing cascading timeouts.
- Disabling referential integrity constraints during bulk ETL imports without post-import consistency validation when configuring Redis Sentinel Split Skipping relational consistency checks risks introducing orphaned records and referential integrity violations into production tables.
- Relying on database snapshot backups that have never been verified through automated restoration drills when configuring Redis Sentinel Split Backups that have never been tested with automated restore verification frequently fail during real disaster recovery scenarios.