Q: How do you implement zero-downtime deployments for a stateful service?
Zero-downtime deployments for a stateful service mean updating one replica at a time while the rest keep serving, so the data tier never loses quorum or the primary.
Want to master this scenario in a live sandbox? KodeKloud's Enterprise GitOps with ArgoCD & Kubernetes Rollouts covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: <strong>Zero-downtime deployments for a stateful service</strong> mean updating one replica at a time while the rest keep serving, so the data tier never loses quorum or the primary. Unlike stateless apps you cannot just surge extra pods: each pod owns identity and data. The answer combines a StatefulSet rolling update, readiness that reflects replication health, a PodDisruptionBudget, graceful shutdown and backward-compatible schema changes. Interviewers ask to see whether you recognise that the data layer, not the container image, is the hard part.
Mental model: why stateful is different
A stateful pod has a stable identity, its own volume, and a role in a replication or consensus group.
- Ordered updates: A StatefulSet with
RollingUpdateupdates pods in reverse ordinal order (db-2, thendb-1, thendb-0) and waits for each to be Ready before continuing. - No surge: There is no extra pod created during the update; capacity temporarily drops by one, so you need enough replicas to tolerate that.
- Data attached: The PersistentVolumeClaim follows the pod name. Restarting is quick only if the volume reattaches quickly.
- Roles matter: Updating a primary first causes a failover. A good rollout updates replicas, then switches over, then updates the old primary.
- Client behaviour: Clients must retry or reconnect during a leader change. Connection poolers and driver retry settings are part of the zero-downtime story.
Hands-on: zero-downtime deployments for a stateful service
Configure the StatefulSet for controlled, canary-style updates.
apiVersion: apps/v1
kind: StatefulSet
metadata: {name: db, namespace: prod}
spec:
replicas: 3
serviceName: db
podManagementPolicy: OrderedReady
updateStrategy:
type: RollingUpdate
rollingUpdate:
partition: 3 # hold all pods; lower to roll out
template:
spec:
terminationGracePeriodSeconds: 120
containers:
- name: db
image: registry.example.com/db:2.1.0
readinessProbe:
exec: {command: ["/bin/sh","-c","/opt/check-replication-lag.sh"]}
lifecycle:
preStop:
exec: {command: ["/bin/sh","-c","/opt/graceful-stepdown.sh"]}
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: {name: db, namespace: prod}
spec:
maxUnavailable: 1
selector: {matchLabels: {app: db}}
- Use partition for canary:
updateStrategy.rollingUpdate.partition: Nupdates only pods with ordinal greater than or equal to N. Lower it step by step. - Readiness must be truthful: A pod should become Ready only after it has caught up with replication, not merely when the port opens.
- Graceful shutdown: A
preStophook andterminationGracePeriodSecondslet the node flush, step down as leader or drain connections. - Protect quorum: A PodDisruptionBudget with
maxUnavailable: 1stops node drains from taking two replicas at once. - Services: Use a headless Service for stable pod DNS plus a separate read or write Service whose selector or endpoints follow the current primary.
Roll out in stages and verify each step
Lower the partition gradually and prove health between steps.
kubectl patch statefulset db -n prod --type merge \
-p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":2}}}}'
kubectl rollout status statefulset/db -n prod
# validate replication, then continue
kubectl patch statefulset db -n prod --type merge \
-p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":0}}}}'
kubectl get pods -n prod -l app=db -o wide
- Canary one replica: Set partition to 2 so only
db-2updates; watch replication lag, errors and latency. - Continue: Set partition to 1, then 0. At each stage confirm
kubectl rollout status statefulset/dbcompletes. - Check the application: Query the cluster's own status (replication state, leader, quorum) rather than relying only on Kubernetes Ready.
- Switch over deliberately: Before updating the current primary, trigger a controlled switchover with your database tooling or operator.
- Pause points: Wait several minutes of steady metrics at each partition value before continuing, because replication problems often surface only under load.
Failure modes and edge cases
Most outages here come from data compatibility and ordering, not Kubernetes.
- Schema changes: Use expand and contract. Deploy additive changes first, then code that uses them, then remove old structures in a later release, so N and N-1 both work.
- Stuck rollout: If a new pod never becomes Ready, a StatefulSet halts and does not roll back automatically; fix forward or revert the image and delete the broken pod.
- RWO volume reattach delay: If a pod moves to another node, the volume must detach and attach first, which can add minutes during node failure.
- Downgrade impossibility: Some storage formats cannot be read by older versions after upgrade, so take a backup or snapshot before the canary and test restore.
- Split brain risk: If readiness is too lenient, a lagging replica can be promoted during failover and lose recent writes; tune probes and failover thresholds together.
Production practice
Automate the safety net and consider an operator.
- Operators: Mature operators for databases handle switchover, version upgrades and backups, and encode the ordering that raw StatefulSets cannot.
- Pre-flight checks: CI should run an upgrade test from the previous version on a staging cluster with real data volume.
- Backups: Take a
VolumeSnapshotor logical backup before each production upgrade and verify restores regularly. - Observability: Alert on replication lag, leader changes and under-replicated partitions during rollout windows.
- Practice failure: Rehearse killing a replica mid-upgrade in staging so the team knows whether the system recovers on its own.
- Zero downtime for a stateful service means updating one replica at a time while quorum and the primary stay available, using a StatefulSet rolling update, PDB and graceful shutdown.
- I verify using replication lag, leader status and rollout status at every partition step, and I take a snapshot first.
- I automate with operators, upgrade tests in CI and expand-and-contract migrations.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Updating the primary first It forces an unplanned failover and write interruption; update replicas and switch over deliberately.
- A readiness probe that only checks the port A pod that has not caught up with replication receives traffic and serves stale or missing data.
- Skipping a PodDisruptionBudget A node drain can evict two replicas at once and break quorum.
- Running a destructive migration in the same release The previous version cannot read the new schema, which removes your ability to roll back.