⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 5 of 207 in Kubernetes
Senior DevOps / SRE Kubernetes Deployment Strategies & Rollbacks Incident Recovery
🎯 Target Role / Context: SRE Incident Management

Q: New deployment breaks production — how would you rollback?

Fast-response production incident runbook for rolling back broken Kubernetes deployments across native kubectl, GitOps (ArgoCD/Flux), and handling database migration traps.

#Kubernetes #Deployment #Rollback #GitOps #ArgoCD #Database Migrations
🎙️ Candidate Opening & Architectural Context
"When a new deployment causes production degradation, the priority is mean time to recovery (MTTR). The rollback path depends on whether you run imperative deployments or declarative GitOps."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: When a new deployment causes production degradation, the priority is mean time to recovery (MTTR). The rollback path depends on whether you run imperative deployments or declarative GitOps.

1️⃣

Immediate Imperative Rollback (kubectl rollout undo)

If using native Kubernetes deployments:

  • kubectl rollout undo deployment/<deployment-name> -n <ns>: Instantly rolls back to the previous revision.
  • kubectl rollout status deployment/<deployment-name>: Monitor the rollback progress in real-time.
  • To target a specific revision: kubectl rollout history deployment/<name> followed by kubectl rollout undo deployment/<name> --to-revision=3.
  • How it works under the hood: Kubernetes points the Deployment back to the previous healthy ReplicaSet, scaling it up while scaling down the broken ReplicaSet.
2️⃣

GitOps Rollback (ArgoCD / Flux Reality)

In GitOps, manual kubectl rollouts will be reverted by self-healing:

  • The GitOps Trap: If you run kubectl rollout undo while ArgoCD has auto-sync + self-heal enabled, ArgoCD will detect drift and immediately re-deploy the broken version!
  • Proper GitOps Rollback: Run git revert HEAD && git push origin main in the manifest repository. ArgoCD syncs and restores the previous commit cleanly.
  • Emergency Fast-Path: In ArgoCD UI/CLI, click Disable Auto-Sync, roll back revision, then fix Git repository.
Advertisement
3️⃣

The Database Migration Trap

Can code be safely rolled back if database schema migrated?

  • If the release included destructive database schema migrations (e.g. dropped a column or renamed a table), rolling back application code will crash the previous version because old code expects the old schema.
  • The Rule: Production deployments must follow Expand and Contract database migrations (additive changes first, deploy code, cleanup in next release).
  • If a migration broke backward compatibility: Coordinate with DBAs to apply a compensating forward migration or restore DB snapshot before reverting code.
4️⃣

Post-Incident & Blameless Post-Mortem

Preventing the same failure in future releases:

  • Conduct blameless post-mortem: Why didn't automated CI tests or staging catch this?
  • Implement Canary Deployments with Argo Rollouts or Flagger: Route 5% traffic to canary; automatically abort if 5xx errors spike without human intervention.
  • Add automated smoke tests and readiness probe validation.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Know your deployment mechanism: In pure K8s, use 'kubectl rollout undo'; in GitOps (ArgoCD), disable auto-sync or 'git revert' to prevent self-heal fighting. Never do destructive DB migrations in single-step releases."
⚡ 60-Second Elevator Pitch Talking Points
  • Immediate action: Run 'kubectl rollout undo deployment/<name>' to revert to previous ReplicaSet.
  • If GitOps (ArgoCD/Flux): Disable auto-sync immediately or 'git revert HEAD && git push' so self-heal doesn't re-break it.
  • Database check: Verify if DB migrations ran. If additive, rollback is safe. If destructive, apply compensating migration.
  • Verify recovery: Monitor 'kubectl rollout status', ALB 5xx metrics, and application logs.
  • Post-mortem: Implement progressive delivery (Argo Rollouts/Canary) with automatic metric-based rollback.
🎬 60-Second Video Walkthrough
Watch the Animated Breakdown & Practice Live

Mastered this procedure? Subscribe to @naveedkumbhar01 on YouTube for daily 60-second SRE incident runbooks, Kubernetes triage shortcuts, and high-impact interview elevator pitches.

🔔 Subscribe on YouTube (Free) ▶ Watch Short (60s) ↗
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Over-granting cluster-admin permissions to pod ServiceAccounts instead of scoping namespace Roles when configuring Deployment Breaks Fast Over-granting cluster-wide permissions violates least-privilege principles and allows compromised pods to escalate privileges across namespaces.
  2. Referencing mutable latest container tags across production Kubernetes Deployment specs when configuring Deployment Breaks Fast Deployments referencing :latest risk inconsistent versions across replica pods and break deterministic rollback capabilities.
  3. Omitting pod readiness probes, causing traffic to hit uninitialized application containers when configuring Deployment Breaks Fast Readiness probes gate traffic delivery; missing probes cause Kubernetes to route user requests to pods before dependencies or caches are loaded.
  4. Executing diagnostic triage without analyzing chronological cluster events and describe outputs when configuring Deployment Breaks Fast Cluster events provide sequential timestamps of scheduler decisions, probe failures, and image pull timeouts essential for rapid root-cause isolation.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →