Q: How do you handle zero-downtime database migrations in a distributed application?
Step-by-step architectural execution of the Expand and Contract pattern to migrate database schemas across distributed microservices with zero downtime.
Want to master this scenario in a live sandbox? KodeKloud's PostgreSQL Database Administration & High Availability Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Phase 1: Expand (Additive Non-Breaking Schema Change)
Add new columns or tables without destructive constraints. Never rename a column directly. If transitioning from `phone` to `phone_number`, add `phone_number` as a nullable column. Run this migration in CI before deploying code.
-- Step 1: Add new column safely
ALTER TABLE customers ADD COLUMN phone_number VARCHAR(32);
Phase 2: Dual-Writing Application Release
Deploy Application Version 2.0. The code writes new data to BOTH `phone` and `phone_number`, but continues reading from `phone`. If a rollback to Version 1.0 is required, Version 1.0 remains 100% operational because `phone` is still updated.
Phase 3: Backfill Historical Data in Batches
Run an asynchronous background worker to copy historical values from `phone` to `phone_number`. Execute backfills in small indexed batches (e.g. 5,000 rows with sleep intervals) to prevent table locking or replication lag.
-- Batch backfill query with sleep to avoid locks
UPDATE customers
SET phone_number = phone
WHERE id BETWEEN 1 AND 5000 AND phone_number IS NULL;
Phase 4 & 5: Switch Read Traffic & Contract (Deprecate Old Column)
Deploy Application Version 2.1 to read exclusively from `phone_number`. Add NOT NULL constraints using `VALIDATE CONSTRAINT` (which avoids table locks in PostgreSQL). In a subsequent sprint, drop the legacy `phone` column once zero services query it.
- Apply the Expand-Contract pattern to ensure database backward compatibility at all times.
- Make all database schema additions non-breaking (nullable columns without defaults).
- Deploy dual-writing application versions so both legacy and modern replicas remain functional.
- Backfill historical records in rate-limited background batches before deprecating old columns.