Q: You have been given a runbook for handling a "database replication lag" incident, but it's out of date. What should a well-structured SRE runbook contain?

A well-structured runbook is the difference between a 5-minute resolution and a 2-hour scramble. It should be written for the worst case: an on-call engineer at 3 AM who has never seen this specific failure before.

• Title and ownership: "Database Replication Lag Remediation" — Owner: @database-team, Last updated: [date]
• Overview: One paragraph describing what this runbook addresses and when to use it.
• Alert trigger: Which monitoring alert triggers this runbook (link to alert definition).
• Impact assessment:
• What breaks if replication lag exceeds 30s? 5min? 30min?
• Which services are affected?
• Customer-facing impact description
• Diagnostic steps (copy-paste commands):

Essential sections:

# Check replication lag
   mysql -e "SHOW SLAVE STATUS\G" | grep Seconds_Behind_Master
   # Check for long-running queries on primary
   mysql -e "SHOW PROCESSLIST" | grep -v Sleep

• Remediation steps: Ordered from least risky to most risky:
• Step 1: Kill long-running queries on primary
• Step 2: Increase replication parallel workers
• Step 3: Failover to standby (with explicit commands)
• Escalation path: Who to call if the runbook doesn't resolve the issue, with contact details.
• Rollback instructions: How to undo each remediation step if it makes things worse.