Q: A developer says "We should alert on average CPU being high". You say "No, that's a symptom. What's the root cause?" Explain the difference between alerting on symptoms vs root causes with a concrete example.
Symptoms are resource metrics (CPU, memory, disk). Root causes are user-facing impacts (errors, latency, requests failing).
Want to master this scenario in a live sandbox? The Linux Foundation's Prometheus Certified Associate (PCA) & Monitoring Labs covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "Logs tell you what happened, metrics tell you where to look, and distributed traces pinpoint the exact slow component. The interviewer is testing: Alert design philosophy, SRE thinking, user impact.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Initial Diagnostics & Root Cause Analysis
Symptoms are resource metrics (CPU, memory, disk). Root causes are user-facing impacts (errors, latency, requests failing).
- Symptom: CPU > 80%
- Root cause: Application latency > 500ms OR error rate > 1%
- A batch job (acceptable, expected, users don't care)
- A memory leak in a thread (critical, users see slow requests)
- A noisy neighbor VM (critical for your app, but your app itself is fine)
Remediation & Permanent Safeguards
Example: You could have high CPU from: Why symptom alerts fail: If you alert on "CPU > 80%", you'll wake on-call for the batch job and ignore the memory leak causing user errors. Alert fatigue makes engineers stop responding to alerts. Root cause alerting: Instead, alert on: These align with what users *actually experience*. If the root cause is firing, you're guaranteed there's a real problem worth waking for. If CPU is high but latency and errors are normal, sleep through it.
- Efficient code using available resources (healthy, no issue)
- "Error rate > 1%" (users are failing)
- "P99 latency > 500ms for 5 min" (user experience degraded)
- "API response 500 errors > 50/min" (app crashed or hung)
- Symptom: CPU > 80%
- Root cause: Application latency > 500ms OR error rate > 1%
- A batch job (acceptable, expected, users don't care)
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Injecting high-cardinality label dimensions (such as user IDs or request UUIDs) into Prometheus metrics when configuring Observability developer says High-cardinality label values cause exponential time-series growth, consuming gigabytes of TSDB memory and crashing Prometheus servers.
- Evaluating application performance solely through arithmetic mean or p50 latency while ignoring tail latency (p99) when configuring Observability developer says Averages hide catastrophic tail latency outages where 1% of high-value user transactions suffer severe timeouts.
- Failing to inject and extract W3C distributed trace context across message brokers and RPC calls when configuring Observability developer says Breaking distributed trace context propagation fragments end-to-end transaction waterfalls into disconnected, un-correlatable trace segments.
- Configuring alert rule evaluation intervals without sufficient duration damping (for: 5m), inducing alert flap when configuring Observability developer says Instant alerts triggered by transient sub-second CPU or latency blips page on-call engineers unnecessarily and dilute response urgency.