⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All CI/CD & GitOps Interview Questions Scenario 89 of 187 in CI/CD & GitOps
Staff SRE / Principal Architect [L3] CI/CD Additional CI/CD Scenarios Staff SRE Scenario [L3]
🎯 Target Role / Context: CI/CD Scenario #80 (L3) · Additional CI/CD Scenarios

Q: Your team manages a stateful backend handling thousands of long-lived WebSockets. The business requires a zero-downtime deployment strategy without dropping active user connections. Is standard Blue/Green deployment the correct mechanism, and how do you implement graceful WebSocket migration?

No. Standard Blue/Green abruptly severs active WebSockets when the load balancer flips traffic. Use Graceful Rolling Degradation with connection draining and client reconnect frames instead.

#CI/CD #Additional CI/CD Scenarios #L3 #DevOps #Automation #Pipelines
🎙️ Candidate Opening & Architectural Context
"In enterprise CI/CD, you cannot rely on abrupt load balancer cutovers for stateful connections. The interviewer is testing WebSocket lifecycle management, connection draining, and client reconnect handling under production load."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's Enterprise GitOps with ArgoCD & Kubernetes Rollouts covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: In enterprise CI/CD, you cannot rely on abrupt load balancer cutovers for stateful connections. The interviewer is testing WebSocket lifecycle management, connection draining, and client reconnect handling under production load.

1️⃣

Why Standard Blue/Green Fails for WebSockets

Standard Blue/Green will drop active sessions. When the load balancer flips targets immediately, TCP connections bound to old instances terminate abruptly, causing an avalanche of simultaneous reconnects.

  • Spin up the new Kubernetes ReplicaSet alongside the existing pods.
  • Configure the Kubernetes Service or Ingress to direct only *new* incoming WebSocket handshake connections to the new Pods.
  • Send a SIGTERM to old Pods. The application catches the signal, stops accepting new requests, and sends an application-level reconnect frame to connected clients.
2️⃣

Graceful Rolling Degradation Architecture

Implement a staggered migration strategy to prevent thundering herds on new instances:

  • Configure preStop hooks with sleep 15 to let the ingress layer remove old pod endpoints before application shutdown.
  • Inject randomized jitter (e.g. 1-10 seconds) into client reconnect logic so thousands of clients do not flood the new backend at once.
  • Set terminationGracePeriodSeconds: 300 (or higher) to give existing WebSockets time to finish active workflows.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Standard Blue/Green abruptly severs long-lived WebSockets. Use graceful connection draining with application-level reconnect frames during rolling deployments."
⚡ 60-Second Elevator Pitch Talking Points
  • Do not use standard Blue/Green for long-lived WebSockets because instantaneous load balancer flipping drops all active connections.
  • Deploy the new ReplicaSet and shift new handshake traffic to the new instances while letting existing connections drain.
  • Dispatch a SIGTERM to old pods to initiate graceful application-level reconnect frames before shutting down.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Promoting application releases to production without an automated, canary-validated rollback mechanism when configuring team heavily tightly Pipelines that lack automated rollback mechanisms force engineering teams to scramble manual hotfixes during severe production regressions.
  2. Running concurrent deployment pipeline jobs without stage mutex locking on shared target environments when configuring team heavily tightly Concurrent pipeline runs on the same environment race to apply state changes, causing out-of-order commits and unpredictable deployment state.
  3. Omitting dependency lockfiles (package-lock.json, go.sum, Cargo.lock) from continuous integration builds when configuring team heavily tightly Omitting lockfiles allows upstream package updates to break builds unpredictably without any changes to application codebase.
  4. Failing to define dynamic cache invalidation keys for CI package managers and layer caches when configuring team heavily tightly Stale build caches mask underlying dependency incompatibilities or pull outdated libraries, while un-cached runs slow feedback loops.
Advertisement
Want more CI/CD & GitOps scenarios?
Explore our complete collection of scenario-based CI/CD & GitOps interview runbooks.
Browse All CI/CD & GitOps Questions →