Q: When measuring API latency, why is an Average (Mean) a terrible metric compared to Percentiles (P95, P99)?
An Average hides extreme outliers. If 99 users experience lightning-fast 10ms latencies, but 1 user hits a database timeout and waits 5,000ms, the mathematical average is ~60ms.
Want to master this scenario in a live sandbox? The Linux Foundation's Prometheus Certified Associate (PCA) & Monitoring Labs covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "Our SRE team tackled this monitoring and metrics bottleneck to eliminate false-positive alert fatigue. The interviewer is testing: Statistical distributions in distributed systems, long-tail latency.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Production Solution & Architecture
An Average hides extreme outliers. If 99 users experience lightning-fast 10ms latencies, but 1 user hits a database timeout and waits 5,000ms, the mathematical average is ~60ms. It looks perfectly healthy on a dashboard, masking the fact that a user had a terrible, broken experience. Percentiles (like P99) order all requests from fastest to slowest. A P99 of 800ms means that 99% of requests were faster than 800ms, and the worst 1% of users experienced 800ms or worse. Alerting on P99 or P99.9 ensures you are monitoring the "long-tail" latency, protecting the experience of your most heavily impacted customers rather than just the majority.
- Immediate Triage: An Average hides extreme outliers. If 99 users experience lightning-fast 10ms latencies, but 1
- Run targeted verification commands before modifying configuration.
- Automate permanent guardrails (CI check, alerts, IaC policy) to prevent recurrence.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Injecting high-cardinality label dimensions (such as user IDs or request UUIDs) into Prometheus metrics when configuring Observability measuring latency High-cardinality label values cause exponential time-series growth, consuming gigabytes of TSDB memory and crashing Prometheus servers.
- Evaluating application performance solely through arithmetic mean or p50 latency while ignoring tail latency (p99) when configuring Observability measuring latency Averages hide catastrophic tail latency outages where 1% of high-value user transactions suffer severe timeouts.
- Failing to inject and extract W3C distributed trace context across message brokers and RPC calls when configuring Observability measuring latency Breaking distributed trace context propagation fragments end-to-end transaction waterfalls into disconnected, un-correlatable trace segments.
- Configuring alert rule evaluation intervals without sufficient duration damping (for: 5m), inducing alert flap when configuring Observability measuring latency Instant alerts triggered by transient sub-second CPU or latency blips page on-call engineers unnecessarily and dilute response urgency.