Q: Your team manages a stateful backend handling thousands of long-lived WebSockets. The business requires a zero-downtime deployment strategy without dropping active user connections. Is standard Blue/Green deployment the correct mechanism, and how do you implement graceful WebSocket migration?
No. Standard Blue/Green abruptly severs active WebSockets when the load balancer flips traffic. Use Graceful Rolling Degradation with connection draining and client reconnect frames instead.
Want to master this scenario in a live sandbox? KodeKloud's Enterprise GitOps with ArgoCD & Kubernetes Rollouts covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: In enterprise CI/CD, you cannot rely on abrupt load balancer cutovers for stateful connections. The interviewer is testing WebSocket lifecycle management, connection draining, and client reconnect handling under production load.
Why Standard Blue/Green Fails for WebSockets
Standard Blue/Green will drop active sessions. When the load balancer flips targets immediately, TCP connections bound to old instances terminate abruptly, causing an avalanche of simultaneous reconnects.
- Spin up the new Kubernetes ReplicaSet alongside the existing pods.
- Configure the Kubernetes Service or Ingress to direct only *new* incoming WebSocket handshake connections to the new Pods.
- Send a
SIGTERMto old Pods. The application catches the signal, stops accepting new requests, and sends an application-level reconnect frame to connected clients.
Graceful Rolling Degradation Architecture
Implement a staggered migration strategy to prevent thundering herds on new instances:
- Configure preStop hooks with
sleep 15to let the ingress layer remove old pod endpoints before application shutdown. - Inject randomized jitter (e.g. 1-10 seconds) into client reconnect logic so thousands of clients do not flood the new backend at once.
- Set
terminationGracePeriodSeconds: 300(or higher) to give existing WebSockets time to finish active workflows.
- Do not use standard Blue/Green for long-lived WebSockets because instantaneous load balancer flipping drops all active connections.
- Deploy the new ReplicaSet and shift new handshake traffic to the new instances while letting existing connections drain.
- Dispatch a SIGTERM to old pods to initiate graceful application-level reconnect frames before shutting down.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Promoting application releases to production without an automated, canary-validated rollback mechanism when configuring team heavily tightly Pipelines that lack automated rollback mechanisms force engineering teams to scramble manual hotfixes during severe production regressions.
- Running concurrent deployment pipeline jobs without stage mutex locking on shared target environments when configuring team heavily tightly Concurrent pipeline runs on the same environment race to apply state changes, causing out-of-order commits and unpredictable deployment state.
- Omitting dependency lockfiles (package-lock.json, go.sum, Cargo.lock) from continuous integration builds when configuring team heavily tightly Omitting lockfiles allows upstream package updates to break builds unpredictably without any changes to application codebase.
- Failing to define dynamic cache invalidation keys for CI package managers and layer caches when configuring team heavily tightly Stale build caches mask underlying dependency incompatibilities or pull outdated libraries, while un-cached runs slow feedback loops.