Q: You have an alerting rule: `CPU Usage > 80% for 5 minutes`. It keeps waking you up at 3 AM for a backend batch processing worker, but it resolves itself after 10 minutes without issues. What do you do?
Waking up for non-actionable, self-resolving alerts creates alert fatigue and burns out engineers. The alert is poorly designed for this workload.
Want to master this scenario in a live sandbox? The Linux Foundation's Prometheus Certified Associate (PCA) & Monitoring Labs covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "In an interview, I explain how we designed actionable, symptom-based alerting using the Four Golden Signals. The interviewer is testing: Alert fatigue reduction, actionable alerting, understanding of batch workloads.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Initial Diagnostics & Root Cause Analysis
Waking up for non-actionable, self-resolving alerts creates alert fatigue and burns out engineers. The alert is poorly designed for this workload.
- Disable or silence the CPU alert for this specific batch worker tier.
- Replace it with a Symptom-Based Alert (SLI/SLO): Alert if the batch job queue age exceeds X minutes or if the job failure rate spikes. Alert on the *outcome* the business cares about, not the resource utilization.
Remediation & Permanent Safeguards
Batch processing workers are *supposed* to use 100% CPU to finish their jobs as quickly as possible. CPU usage is not a symptom of failure here; it's a measure of efficiency. I would:
- Disable or silence the CPU alert for this specific batch worker tier.
- Replace it with a Symptom-Based Alert (SLI/SLO): Alert if the batch job queue age exceeds X minut...
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Injecting high-cardinality label dimensions (such as user IDs or request UUIDs) into Prometheus metrics when configuring Observability alerting rule High-cardinality label values cause exponential time-series growth, consuming gigabytes of TSDB memory and crashing Prometheus servers.
- Evaluating application performance solely through arithmetic mean or p50 latency while ignoring tail latency (p99) when configuring Observability alerting rule Averages hide catastrophic tail latency outages where 1% of high-value user transactions suffer severe timeouts.
- Failing to inject and extract W3C distributed trace context across message brokers and RPC calls when configuring Observability alerting rule Breaking distributed trace context propagation fragments end-to-end transaction waterfalls into disconnected, un-correlatable trace segments.
- Configuring alert rule evaluation intervals without sufficient duration damping (for: 5m), inducing alert flap when configuring Observability alerting rule Instant alerts triggered by transient sub-second CPU or latency blips page on-call engineers unnecessarily and dilute response urgency.