Q: You need to do a zero-downtime migration of a StatefulSet (e.g., upgrading Postgres version). Walk me through your approach.
StatefulSets don't support zero-downtime rolling updates as cleanly as Deployments because each pod has unique state.
#Kubernetes #Advanced Scenarios #L3 #Container Orchestration #K8s #Terraform State
🎙️ Candidate Opening & Architectural Context
""In one of our high-traffic production clusters, our SRE team handled this incident using a standardized runbook. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix.""
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Initial Diagnostics & Root Cause Analysis
StatefulSets don't support zero-downtime rolling updates as cleanly as Deployments because each pod has unique state.
- Take a backup first — always. Snapshot the PVC with VolumeSnapshot or pg_dump.
- Set
updateStrategytoOnDelete— this lets you control which pods update manually. - Update the StatefulSet spec (new image version).
- Delete pods one at a time — start with replicas (highest ordinal), not the primary. Let each pod restart with the new version and come up healthy before proceeding.
2️⃣
Remediation & Permanent Safeguards
Approach: For major Postgres version upgrades: consider blue-green approach — spin up new StatefulSet, replicate data, cut traffic over.
- Verify replication health between each pod update.
- Update the primary last — failover to a replica first if needed.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Take a backup first — always. Snapshot the PVC with VolumeSnapshot or pg_dump.."
⚡ 60-Second Elevator Pitch Talking Points
- Take a backup first — always. Snapshot the PVC with VolumeSnapshot or pg_dump.
- Set updateStrategy to OnDelete — this lets you control which pods update manually.
- Update the StatefulSet spec (new image version).
Advertisement