Q: How do you perform a Helm rollback?
Technical deep dive into 'helm rollback': how Helm stores release history in Kubernetes Secrets, three-way merge patching, and the critical limitations around CRDs and database schema migrations.
#Helm #Rollback #helm rollback #Release Secrets #CRD #Database
🎙️ Candidate Opening & Architectural Context
"A Helm rollback is not just 're-running old YAML'. It is a precise historical reconciliation against Helm's release metadata stored inside cluster Secrets."
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Step 1: Inspect History & Trigger Rollback
Standard CLI operational runbook:
helm history <release-name> -n <namespace>: Lists all previous revisions, dates, statuses (DEPLOYED, FAILED), and descriptions.- Identify the last known healthy revision (e.g. revision 4).
helm rollback <release-name> 4 -n <namespace> --wait --timeout 3m: Rolls back to revision 4 and waits for pods to pass readiness probes.
2️⃣
Step 2: What Happens Under the Hood?
How Helm executes the rollback internally:
- Release Secrets: Helm stores every revision as a compressed, base64-encoded Secret in the release namespace (e.g.
sh.helm.release.v1.payment-svc.v4). - Three-Way Merge: Helm calculates a three-way merge patch between the current live cluster state, the target revision manifest, and the old manifest.
- New Revision Increment: Rolling back to revision 4 does NOT delete revision 5. It creates Revision 6 whose content matches Revision 4. History is strictly append-only.
3️⃣
Step 3: Critical Limitations & The Database Trap
What senior engineers know that juniors miss:
- Helm NEVER Touches CRDs: CustomResourceDefinitions in the
crds/directory are never upgraded or rolled back by Helm by design (to protect existing custom resources from deletion). - The Database Schema Trap: Helm only rolls back Kubernetes resources (Deployments, ConfigMaps). It cannot roll back a database migration executed by a Helm pre-install job. Application code must be backward-compatible with the schema.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Helm rollback creates a brand new revision that matches the target historical revision manifest using a 3-way merge against cluster release Secrets. Be aware that Helm never rolls back CRDs or database state."
⚡ 60-Second Elevator Pitch Talking Points
- Check history: 'helm history <release> -n <ns>' to find the last healthy revision number.
- Roll back: 'helm rollback <release> <revision_number> --wait'.
- Under the hood: Helm reads the compressed release Secret (sh.helm.release.v1.*), computes a 3-way strategic merge patch, and creates a NEW incremented revision.
- Limitation 1: Helm does not manage or roll back CRDs.
- Limitation 2: Database migrations are not rolled back by Helm; migrations must be backward-compatible.
Advertisement