Q: Your CI/CD pipeline deploys directly to production on every merge to `main`. One Friday afternoon, a developer merges a PR and goes offline. The deployment silently fails halfway through — 30% of pods are running new code, 70% are running old code. There are no automated rollback hooks. How do you recover, and how do you prevent this situation architecturally?

Immediate recovery:

• Identify current state: kubectl rollout status deployment/api — if Progressing is stuck, the rollout is hanging.
• Manually roll back: kubectl rollout undo deployment/api — Kubernetes reverts to the previous ReplicaSet revision immediately. Verify with kubectl rollout history deployment/api.
• --atomic on Helm, or rollout analysis on Argo Rollouts — if the deployment does not reach 100% healthy within a timeout, it automatically rolls back.

Architectural prevention:

• Deployment freeze rules: Block merges to main after 3 PM on Fridays via a GitHub branch protection check that queries the current time. Simple but highly effective.
• Require on-call acknowledgement: The pipeline sends a Slack message with "Deployment starting, confirm you're watching" and waits 60 seconds for a reaction emoji before proceeding. If no reaction, the pipeline pauses and pages on-call.