Q: Your mobile trading and chat application must maintain 5,000,000 simultaneous persistent WebSocket connections to deliver instant price updates and notifications with sub-100ms latency. Edge connections frequently terminate due to mobile network handoffs, triggering reconnect storms. How do you design the networking, gateway tier, pub/sub broker, and Linux kernel to handle 5 million connections reliably?
Engineering a real-time push notification service sustaining 5,000,000 concurrent persistent WebSocket connections with Redis Pub/Sub, horizontal connection gateway pods, and Linux kernel epoll tuning.
Want to master this scenario in a live sandbox? The Linux Foundation's FinOps Certified Practitioner (FOCP) Program covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Maintaining 5 million concurrent stateful WebSocket connections exhausts Linux file descriptors, memory buffers, and breaks standard stateless load balancers. We engineered a massive real-time WebSocket platform utilizing an epoll-based Go gateway tier, Redis Pub/Sub sharding, and Linux socket tuning.
Tune Linux Kernel & OS Limits for Millions of Sockets
Prevent TCP socket memory starvation and file descriptor limits on worker nodes:
- File Descriptors: Increased system limits:
fs.file-max = 10000000and process limits:ulimit -n 1048576. - TCP Memory Tuning: Tuned TCP buffer windows:
net.ipv4.tcp_rmem = 4096 87380 16777216andnet.ipv4.tcp_wmem = 4096 65536 16777216, setting minimum read/write buffers to 4KB to pack 100,000 connections per 16 GB RAM node. - Ephemeral Ports: Configured
net.ipv4.ip_local_port_range = 1024 65535and tunednet.core.somaxconn = 65535to absorb massive connection surges.
Deploy Horizontal Gateway Cluster using epoll / kqueue (Go / Rust)
Manage millions of concurrent connections with minimal CPU context switching:
- Non-Blocking I/O: Built gateway servers in Go using
gobwas/wsor Rust usingtokio-tungstenitewith native Linuxepollevent loops. - Node Capacity: Deployed a cluster of 50 gateway pods (each holding 100,000 connections), distributed behind a Layer 4 Network Load Balancer (AWS NLB) with Maglev consistent hashing.
Route Targeted Messages via Sharded Redis Pub/Sub Mesh
Locate which gateway pod holds the target user's active socket connection:
- Connection Registry: When user
user_4821connects, gateway registers mapping in Redis:SET connection:user_4821 'gateway-pod-14' EX 300(refreshed via heartbeat). - Sharded Redis Pub/Sub: When a notification arrives, publisher checks the registry and publishes the message to the specific pod's channel:
gateway-pod-14. - Direct Emission: Gateway pod 14 receives the event from Redis and writes the JSON packet directly to the user's active WebSocket in < 4 milliseconds.
Mitigate Reconnect Storms via Exponential Backoff & Jitter
Prevent thundering herd collapse when cell towers or wifi networks drop:
- Client Reconnection Jitter: Mobile client libraries enforce randomized exponential backoff:
sleep = min(cap, base * 2^attempt) + rand(0, 1000ms). - Heartbeat Ping/Pong: Gateway sends WebSocket PING every 30 seconds; if client fails to respond with PONG within 10 seconds, the dead socket descriptor is cleaned up immediately.
- Validation Load Test: Sustained 5.2 million live concurrent connections under load with p99 delivery latency of 18ms.
- Tune Linux TCP socket memory buffers down to 4KB to pack 100k connections per node.
- Deploy non-blocking epoll gateway pods behind L4 Network Load Balancers.
- Route targeted notifications through a sharded Redis Pub/Sub cluster in < 5ms.
- Enforce randomized client exponential backoff and jitter to survive massive reconnect storms.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Engineering bespoke in-house developer tools instead of adopting standardized platform Golden Paths when configuring System Design Real Bespoke internal tools create long-term maintenance debt and fragmented workflows, whereas standardized Golden Paths accelerate delivery.
- Overlooking infrastructure cost visibility and FinOps allocation tagging across multi-tenant environments when configuring System Design Real Without granular resource tagging, engineering organizations cannot attribute cloud infrastructure costs to specific teams or business units.
- Failing to define explicit Service Level Objectives (SLOs) and Error Budgets to balance feature velocity with reliability when configuring System Design Real Without agreed SLOs, engineering and product teams lack an objective data-driven framework to prioritize technical debt over new features.
- Neglecting automated continuous drift detection between declarative configuration and deployed cloud state when configuring System Design Real in system-design-high-volume-websocket-push-notification-service Unmanaged drift accumulates over time, causing future automated deployments to fail or overwrite emergency manual adjustments.