Q: A production DB migration succeeds, but the API deployment fails and the DB change isn't backward-compatible. How would you recover safely?
Emergency incident triage and architectural prevention when a database schema migration executes in production, but the API deployment crashes and the schema change cannot be trivially rolled back.
🛠️ Production Runbook & Step-by-Step Resolution
Immediate Triage: Diagnose API Crash & Avoid Destructive Rollback
Do NOT blindly rollback the database if users have already written new transactions, as rolling back could drop columns containing live customer data. Inspect the API boot crash logs to identify the exact schema mismatch.
Apply Forward Compatibility Hotfix / Database Bridge View
If a column was renamed, create an alias view or database trigger that maps old column names to new column names, allowing the previous v1 API to continue operating against the v2 schema while the v2 API bug is patched.
-- Creating an alias column or view for temporary backward compatibility
ALTER TABLE users ADD COLUMN legacy_username VARCHAR(255) GENERATED ALWAYS AS (email) STORED;Enforce the Expand/Contract (Parallel Run) Migration Pattern
Mandate that no deployment may ever execute destructive schema changes in Phase 1. Follow the 3-phase rule: 1) Expand (add new columns, dual-write), 2) Deploy app, 3) Contract (drop old columns in a subsequent release).
- Do not execute destructive database rollbacks that risk dropping live customer data.
- Create a database compatibility bridge (views, triggers, or default constraints) to support the v1 API.
- Push an emergency roll-forward hotfix to resolve the API startup exception.
- Enforce the Expand/Contract deployment pattern: all database migrations must be 100% backward-compatible.