Q: Your high-throughput read API serves 300,000 requests per second. Querying central Redis for every request saturates Redis network bandwidth (10 Gbps limit). Storing a local L1 cache inside each Go/Java application pod delivers 0.01ms latency, but pods serve stale data when entities are updated in the database. How do you design a multi-tier cache hierarchy with sub-millisecond distributed cache invalidation across 200 application pods?
Engineering a high-performance two-tier cache hierarchy (L1 in-memory application cache + L2 shared Redis cluster) handling 300,000 RPS with sub-millisecond invalidation broadcasts via Redis Pub/Sub.
Want to master this scenario in a live sandbox? KodeKloud's PostgreSQL Database Administration & High Availability Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Centralized caches hit network bandwidth scaling limits under massive read loads. We architected a two-tier caching platform combining local L1 in-memory pod caching (Go Ristretto / Java Caffeine), shared L2 Redis Cluster, and distributed invalidation broadcasting over Redis Pub/Sub.
Architect Two-Tier Cache Topology: L1 In-Memory + L2 Redis Cluster
Achieve nanosecond read latency while offloading central Redis network traffic:
- Tier 1 (L1 In-Memory): Each application pod allocates 500 MB RAM to a local in-memory cache (Caffeine for JVM / Ristretto for Go) using TinyLFU eviction policies.
- Tier 2 (L2 Shared Cache): Shared multi-AZ Redis Cluster caching complete serialized entities with a 24-hour TTL.
- Read Flow: Application checks L1 (0.005ms) -> if miss, checks L2 Redis (1.2ms) -> if miss, queries PostgreSQL database (15ms), populating both L2 and L1.
Broadcast Sub-Millisecond Invalidation via Redis Pub/Sub
Evict stale local pod memory the instant an update occurs:
- Write Flow: When an update modifies
product:1234, the writing pod updates PostgreSQL, deletes the key from L2 Redis, and publishes an invalidation event:PUBLISH cache:invalidate 'product:1234'. - Pod Subscription: All 200 application pods maintain an active subscription to
cache:invalidate. - Local Eviction: Upon receiving the broadcast message, every pod immediately purges
product:1234from its local L1 memory in < 1 millisecond.
Mitigate Pub/Sub Packet Drops via Monotonic Versioning & Short TTL
Prevent silent data desynchronization if a pod temporarily loses network connection:
- Short L1 TTL: All L1 cache entries enforce a maximum background TTL of 60 seconds (even if never invalidated), bounding maximum staleness to 60s under catastrophic network splits.
- Monotonic Entity Versioning: Objects embed an incremental version counter:
{ 'id': 1234, 'version': 48 }. If an L1 cache entry has a version lower than an incoming write request header, the local entry is evicted immediately.
Prevent Cache Stampedes via SingleFlight Mutex Suppression
Shield the underlying database during sudden cache eviction spikes:
- SingleFlight Pattern: When an L1/L2 cache miss occurs for a popular key, the pod uses Go
singleflight.Group(or Java synchronized loader) to ensure that only ONE worker thread queries the database. - Shared Result: All 500 concurrent in-flight requests wait for the single database query to complete and share the result in memory.
- Database Impact: Database CPU utilization during cache invalidations dropped from 95% spikes down to a steady 12%.
- Deploy L1 in-memory pod caching (TinyLFU) backed by an L2 Redis Cluster to slash network bandwidth.
- Broadcast real-time cache invalidation messages across all pods in < 1ms via Redis Pub/Sub.
- Enforce short 60s L1 safety TTLs and monotonic entity versioning to handle network blips.
- Use SingleFlight request coalescing to eliminate thundering herd database stampedes.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Applying exclusive-lock DDL table migrations on live production databases during peak traffic windows when configuring System Design Multi DDL migrations that acquire an exclusive table lock block all concurrent queries, queuing connections until the database connection pool exhausts.
- Allowing high-frequency queries to trigger full table scans without appropriate composite indexing when configuring System Design Multi Sequential table scans saturate database CPU and disk I/O, evicting frequently queried data from the buffer cache and causing cascading timeouts.
- Disabling referential integrity constraints during bulk ETL imports without post-import consistency validation when configuring System Design Multi Skipping relational consistency checks risks introducing orphaned records and referential integrity violations into production tables.
- Relying on database snapshot backups that have never been verified through automated restoration drills when configuring System Design Multi Backups that have never been tested with automated restore verification frequently fail during real disaster recovery scenarios.