Q: Your SLO is 99.9% availability. Your current availability for the month is 99.95%. A development team wants to push a massive refactor on Friday evening that hasn't been tested thoroughly. What do you do?
SRE uses Error Budgets to make data-driven decisions between feature velocity and reliability, removing the emotion from the conversation.
Want to master this scenario in a live sandbox? The Linux Foundation's Prometheus Certified Associate (PCA) & Monitoring Labs covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "Our SRE team tackled this monitoring and metrics bottleneck to eliminate false-positive alert fatigue. The interviewer is testing: Error budgets, SRE cultural practices, blameless decision making.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Production Solution & Architecture
SRE uses Error Budgets to make data-driven decisions between feature velocity and reliability, removing the emotion from the conversation. With a 99.9% SLO, we are allowed a 0.1% error budget for the month. Since we are at 99.95%, we have positive error budget remaining. Technically, they have the budget to deploy. However, Friday evening deployments violate the core risk mitigation practice of having support available during business hours. I would advise them: "You have the error budget, but deploying Friday night risks a major outage over the weekend. Because an outage will burn through the remaining budget—halting all feature deployments next week if we drop below 99.9%—I strongly recommend waiting until Monday morning when the team can monitor the rollout safely."
- Immediate Triage: SRE uses Error Budgets to make data-driven decisions between feature velocity and reliability,
- Run targeted verification commands before modifying configuration.
- Automate permanent guardrails (CI check, alerts, IaC policy) to prevent recurrence.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Injecting high-cardinality label dimensions (such as user IDs or request UUIDs) into Prometheus metrics when configuring Observability 999% availability High-cardinality label values cause exponential time-series growth, consuming gigabytes of TSDB memory and crashing Prometheus servers.
- Evaluating application performance solely through arithmetic mean or p50 latency while ignoring tail latency (p99) when configuring Observability 999% availability Averages hide catastrophic tail latency outages where 1% of high-value user transactions suffer severe timeouts.
- Failing to inject and extract W3C distributed trace context across message brokers and RPC calls when configuring Observability 999% availability Breaking distributed trace context propagation fragments end-to-end transaction waterfalls into disconnected, un-correlatable trace segments.
- Configuring alert rule evaluation intervals without sufficient duration damping (for: 5m), inducing alert flap when configuring Observability 999% availability Instant alerts triggered by transient sub-second CPU or latency blips page on-call engineers unnecessarily and dilute response urgency.