⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All FinOps & System Design Interview Questions Scenario 40 of 44 in FinOps & System Design
Staff Distributed Systems Architect System Design Distributed Systems Consensus & SRE System Design
🎯 Target Role / Context: Principal Distributed Systems Architect / Staff SRE System Design Interview

Q: Your distributed payment system requires a distributed lock to prevent double-spending when processing credit card charges. A developer implemented a simple Redis distributed lock (SET NX EX). During a JVM Stop-The-World garbage collection pause, the lock expired, another worker acquired the lock, and two workers concurrently charged the user's card. How do you design a mathematically sound Distributed Lock Manager that is immune to GC pauses and network partitions?

Engineering a mathematically correct distributed lock manager for mission-critical financial transactions, analyzing Redlock timing vulnerabilities, and implementing etcd/ZooKeeper leases with monotonic fencing tokens.

#System Design #Distributed Locks #Redlock #ZooKeeper #etcd #Consensus #Fencing Tokens #Comparisons
🎙️ Candidate Opening & Architectural Context
"Simple Redis distributed locks (and even multi-node Redlock) are vulnerable to timing assumptions: asynchronous clock drift, network delays, and GC pauses can cause two clients to simultaneously hold the 'exclusive' lock. We architected a mathematically correct Distributed Lock Manager using etcd leases and monotonic Fencing Tokens."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? The Linux Foundation's FinOps Certified Practitioner (FOCP) Program covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: Simple Redis distributed locks (and even multi-node Redlock) are vulnerable to timing assumptions: asynchronous clock drift, network delays, and GC pauses can cause two clients to simultaneously hold the 'exclusive' lock. We architected a mathematically correct Distributed Lock Manager using etcd leases and monotonic Fencing Tokens.

1️⃣

Analyze Distributed Lock Failure Modes: Martin Kleppmann vs Redlock

Understand why time-based locks fail in asynchronous distributed systems:

  • The GC Pause Flaw: Client 1 acquires lock with 10s TTL. Client 1 hits a 15s JVM GC pause. The lock expires. Client 2 acquires the lock. Client 1 wakes up and executes its write concurrently with Client 2, corrupting shared data.
  • Clock Drift Vulnerability: NTP adjustments on individual nodes can cause TTLs to expire prematurely.
  • Architectural Conclusion: High-safety locks require consensus systems (etcd / ZooKeeper) paired with storage-layer verification.
Pro Tip: No distributed lock algorithm can guarantee mutual exclusion on the client side in an asynchronous network without downstream storage cooperation.
2️⃣

Deploy Strong-Consistency Consensus Lock Engine via etcd v3 Leases

Leverage Raft linearizable consensus for exclusive lock ownership:

  • etcd Leases: Client requests a Lease with 10-second TTL: lease = client.Grant(context.Background(), 10).
  • Atomic Key Creation: Client executes transaction creating lock key: txn.If(CreateRevision(key) == 0).Then(Put(key, clientID, WithLease(lease))).
  • Keep-Alive Heartbeat: A background goroutine continuously sends KeepAlive heartbeats; if the client crashes or loses network connectivity, etcd automatically deletes the key upon lease expiry.
Pro Tip: etcd Raft consensus guarantees that exactly ONE client can create the lock key at any instant, backed by linearizable read/write quorums.
Advertisement
3️⃣

Enforce Monotonic Fencing Tokens at the Database Storage Layer

Defeat the GC pause problem by validating tokens on write operations:

  • Monotonic Revision Token: Whenever etcd grants a lock, it returns the global Raft revision number (e.g. token = 48291). Each subsequent lock grant receives a strictly higher integer (48292).
  • Storage Guardrail: When writing to the database, client passes the fencing token. The database executes: UPDATE accounts SET balance = balance - 100, last_token = 48292 WHERE account_id = 'A' AND last_token < 48292.
  • Stale Write Rejection: If Client 1 wakes up from a GC pause with stale token 48291, the database rejects the write because last_token (48292) >= 48291.
Pro Tip: Fencing tokens push validation to the storage resource, mathematically guaranteeing that delayed requests from former lock holders are rejected.
4️⃣

Implement Event-Driven Lock Waiters via etcd Watch API

Eliminate CPU-wasting busy-wait polling loops:

  • Watch Stream: Contending workers do not poll etcd in a loop; they establish an etcd Watch on the lock key.
  • Instant Notification: When the active lock holder finishes and releases the key, etcd pushes an event stream notification, waking the next waiter in milliseconds.
  • Verification Benchmark: Executed 100,000 concurrent lock competitions with simulated 10-second thread pauses; zero duplicate writes occurred.
Pro Tip: The etcd Watch API provides instantaneous lock handover without wasting network bandwidth on aggressive polling.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"A mathematically sound distributed lock requires strong consensus (etcd/Raft) for lock leases paired with strictly increasing monotonic fencing tokens validated at the storage layer to completely prevent GC-pause race conditions."
⚡ 60-Second Elevator Pitch Talking Points
  • Acknowledge that client-side locks cannot guarantee safety during GC pauses without storage checks.
  • Use etcd v3 linearizable Raft consensus with KeepAlive leases for distributed lock acquisition.
  • Issue strictly increasing monotonic fencing tokens with every lock grant.
  • Enforce conditional database updates (WHERE last_token < current_token) to reject stale writes.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Engineering bespoke in-house developer tools instead of adopting standardized platform Golden Paths when configuring System Design Distributed Bespoke internal tools create long-term maintenance debt and fragmented workflows, whereas standardized Golden Paths accelerate delivery.
  2. Overlooking infrastructure cost visibility and FinOps allocation tagging across multi-tenant environments when configuring System Design Distributed Without granular resource tagging, engineering organizations cannot attribute cloud infrastructure costs to specific teams or business units.
  3. Failing to define explicit Service Level Objectives (SLOs) and Error Budgets to balance feature velocity with reliability when configuring System Design Distributed Without agreed SLOs, engineering and product teams lack an objective data-driven framework to prioritize technical debt over new features.
  4. Neglecting automated continuous drift detection between declarative configuration and deployed cloud state when configuring System Design Distributed in system-design-distributed-lock-manager-consensus Unmanaged drift accumulates over time, causing future automated deployments to fail or overwrite emergency manual adjustments.
Advertisement
Want more FinOps & System Design scenarios?
Explore our complete collection of scenario-based FinOps & System Design interview runbooks.
Browse All FinOps & System Design Questions →