Q: Your production DB needs a schema migration that could lock tables for minutes. How do you do this with zero downtime?
Use a no-lock migration approach: add new column (no lock), backfill data in batches, add new index concurrently, switch app to use new c...
#AWS #Cost & Architecture #L3 #Cloud #Infrastructure
🎙️ Candidate Opening & Architectural Context
""In a previous role, our monitoring paged me for a similar incident across our AWS VPC infrastructure. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix.""
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Production Solution & Architecture
Use a no-lock migration approach: add new column (no lock), backfill data in batches, add new index concurrently, switch app to use new column, drop old column later. For major migrations, use tools like gh-ost (MySQL) or pg_repack (Postgres).
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Use a no-lock migration approach: add new column (no lock), backfill data in batches, add new index concurrently, switch app to us."
⚡ 60-Second Elevator Pitch Talking Points
- Immediate Triage: Use a no-lock migration approach: add new column (no lock), backfill data in batches, add new i
- Run targeted verification commands before modifying configuration.
- Automate permanent guardrails (CI check, alerts, IaC policy) to prevent recurrence.
Advertisement