Q: Your global SaaS platform requires sub-10ms write latency for users in both the US and Europe with 99.999% availability. You need to enable Azure Cosmos DB Multi-Region Multi-Write. How do you design conflict resolution, select consistency levels, and configure client SDK connection policies to avoid stale reads and split-brain data loss?
Engineering a globally distributed active-active data tier using Azure Cosmos DB Multi-Region Multi-Write, conflict resolution policies (LWW vs Stored Procedures), and session consistency tuning.
#Azure #Cosmos DB #Multi-Region #Multi-Write #NoSQL #SRE
🎙️ Candidate Opening & Architectural Context
"Previously, our Cosmos DB instance ran single-region writes in East US with read replicas in West Europe. European users suffered 120ms write latency across the Atlantic. We migrated to Multi-Region Multi-Write with custom Last-Write-Wins conflict handling."
Advertisement
⚡ Recommended Practice Lab
Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Enable Multi-Region Multi-Write on Cosmos DB Account
Reconfigure the account topology to permit write operations across all regional replicas:
- CLI Provisioning: Executed
az cosmosdb update --name cosmos-global --resource-group rg-data --enable-multiple-write-locations true --locations regionName=eastus failoverPriority=0 isZoneRedundant=true --locations regionName=westeurope failoverPriority=1 isZoneRedundant=true. - SLA Upgrade: Multi-region multi-write elevates Cosmos DB availability SLA to 99.999% for both reads and writes.
Pro Tip: Enabling multiple write locations allows write operations to complete against the local datacenter, reducing write latency from >100ms down to <8ms.
2️⃣
Configure Conflict Resolution: Last-Write-Wins (LWW) vs Custom
Establish deterministic data resolution rules when simultaneous concurrent writes update the same document in different regions:
- LWW Policy: Configured
conflictResolutionPolicy: { mode: 'LastWriteWins', conflictResolutionPath: '/_ts' }for standard document updates. - Custom Stored Procedure: For financial balance adjustments, deployed a Javascript Conflict Resolution Stored Procedure (SP) that executes vector-clock merge logic rather than discarding concurrent updates.
Pro Tip: Default Last-Write-Wins relies on the highest timestamp. If concurrent writes hit US and EU simultaneously, the earlier write is discarded unless custom merge logic is defined.
3️⃣
Select Consistency Level: Session vs Bounded Staleness
Balance CAP theorem trade-offs between write latency and read guarantees:
- Session Consistency: Selected Session Consistency for user-facing workloads, ensuring monotonic reads and read-your-own-writes guarantees within the client session token.
- Session Token Forwarding: Implemented HTTP response header passing of
x-ms-session-tokenbetween distributed microservices to ensure read consistency across services.
Pro Tip: Strong consistency is not supported with Multi-Region Multi-Write because synchronous cross-region consensus violates low-latency physics.
4️⃣
Configure Azure Cosmos DB SDK Connection Policies
Ensure application SDK instances automatically route to their local geographic region:
- Preferred Regions List: Configured SDK client builder:
CosmosClientBuilder.endpoint(URI).preferredRegions(List.of('East US', 'West Europe')).buildClient(). - Dynamic Fallback: Verified that during a simulated datacenter outage, SDK automatically retried writes against West Europe within 200ms.
Pro Tip: Without configuring preferredRegions in the SDK client, applications may inadvertently route writes across oceans to distant regions.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Cosmos DB Multi-Region Multi-Write achieves sub-10ms global writes and 99.999% SLA by combining local writes, Session consistency tokens, and well-defined conflict resolution policies."
⚡ 60-Second Elevator Pitch Talking Points
- Enable multiple write locations on Cosmos DB account across geographic regions.
- Implement Last-Write-Wins or custom Stored Procedure conflict resolution policies.
- Use Session Consistency with session token propagation to guarantee monotonic reads.
- Configure SDK preferredRegions list to ensure applications write to the closest datacenter.
Advertisement