Q: Every alert in your system has severity `critical`, so on-call gets paged for low-risk issues. How do you design alert severity levels?
Severity should map to required human action.
Want to master this scenario in a live sandbox? The Linux Foundation's Prometheus Certified Associate (PCA) & Monitoring Labs covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "During a high-traffic production event, our observability stack proved essential in isolating this latency surge. The interviewer is testing: Alert prioritization, paging discipline, incident response.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Initial Diagnostics & Root Cause Analysis
Severity should map to required human action.
- Page immediately: User-facing outage, fast error budget burn, data loss risk, security-impacting production issue.
- Urgent ticket or chat: Degradation that needs same-day action but is not actively hurting users.
- Backlog ticket: Capacity trend, cleanup task, non-production issue, or informational warning.
Remediation & Permanent Safeguards
I would define levels like: Each alert should include owner, service, impact, runbook, dashboard link, and escalation path. If no one needs to wake up and act immediately, it should not be a paging alert. This reduces alert fatigue and makes critical pages meaningful again.
- Page immediately: User-facing outage, fast error budget burn, data loss risk, security-impacting ...
- Urgent ticket or chat: Degradation that needs same-day action but is not actively hurting users.
- Backlog ticket: Capacity trend, cleanup task, non-production issue, or informational warning.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Injecting high-cardinality label dimensions (such as user IDs or request UUIDs) into Prometheus metrics when configuring Observability Every alert High-cardinality label values cause exponential time-series growth, consuming gigabytes of TSDB memory and crashing Prometheus servers.
- Evaluating application performance solely through arithmetic mean or p50 latency while ignoring tail latency (p99) when configuring Observability Every alert Averages hide catastrophic tail latency outages where 1% of high-value user transactions suffer severe timeouts.
- Failing to inject and extract W3C distributed trace context across message brokers and RPC calls when configuring Observability Every alert Breaking distributed trace context propagation fragments end-to-end transaction waterfalls into disconnected, un-correlatable trace segments.
- Configuring alert rule evaluation intervals without sufficient duration damping (for: 5m), inducing alert flap when configuring Observability Every alert Instant alerts triggered by transient sub-second CPU or latency blips page on-call engineers unnecessarily and dilute response urgency.