Q: An alerting rule fires every 3 seconds, then clears every 5 seconds, creating 150+ PagerDuty incidents per hour. The actual metric oscillates around the threshold. How do you stabilize this?
This is alert flapping—when a metric oscillates around the threshold, causing rapid alert cycles. Engineers ignore the notifications (alert fatigue), defeating their purpose.
Want to master this scenario in a live sandbox? The Linux Foundation's Prometheus Certified Associate (PCA) & Monitoring Labs covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "In an interview, I explain how we designed actionable, symptom-based alerting using the Four Golden Signals. The interviewer is testing: Alert flapping, dampening strategies, alert fatigue reduction.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Initial Diagnostics & Root Cause Analysis
This is alert flapping—when a metric oscillates around the threshold, causing rapid alert cycles. Engineers ignore the notifications (alert fatigue), defeating their purpose.
- Raise the evaluation window: Instead of
cpu > 80%, useavg(cpu) over 5m > 80%. Oscillations within minutes won't trigger; only sustained issues will. - Hysteresis (two-threshold approach):
- Alert fires when metric > 85% (high threshold)
- Alert clears only when metric < 75% (low threshold)
- This creates a "dead zone" between 75-85%, preventing flapping.
Remediation & Permanent Safeguards
Solutions (in order of increasing sophistication): Best practice example (Prometheus):
- alert: HighCPU
expr: rate(node_cpu[1m]) > 0.8
for: 5m # Must be high for 5 consecutive minutes
annotations:
summary: "CPU sustained above 80%"
- Prometheus: Use
for: 5m(must exceed threshold for 5 minutes before triggering). - Aggregation: Instead of single-instance CPU, alert on
avg(cpu) across all instances > 80%. Aggregate metrics are smoother. - Dynamic thresholding: Replace static 80% with
predict_linear(cpu[1h], 3600) > 90%. Only alert if the metric will hit 90% within the hour (giving lead time instead of flapping).
- Raise the evaluation window: Instead of cpu > 80%, use avg(cpu) over 5m > 80%. Oscillations withi...
- Hysteresis (two-threshold approach):
- Alert fires when metric > 85% (high threshold)
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Injecting high-cardinality label dimensions (such as user IDs or request UUIDs) into Prometheus metrics when configuring Observability alerting rule High-cardinality label values cause exponential time-series growth, consuming gigabytes of TSDB memory and crashing Prometheus servers.
- Evaluating application performance solely through arithmetic mean or p50 latency while ignoring tail latency (p99) when configuring Observability alerting rule Averages hide catastrophic tail latency outages where 1% of high-value user transactions suffer severe timeouts.
- Failing to inject and extract W3C distributed trace context across message brokers and RPC calls when configuring Observability alerting rule Breaking distributed trace context propagation fragments end-to-end transaction waterfalls into disconnected, un-correlatable trace segments.
- Configuring alert rule evaluation intervals without sufficient duration damping (for: 5m), inducing alert flap when configuring Observability alerting rule in observability-q042-an-alerting-rule-fires-every-3-seco Instant alerts triggered by transient sub-second CPU or latency blips page on-call engineers unnecessarily and dilute response urgency.