Q: An SRE says the infrastructure is stable, the dev team says the system is slow, and monitoring dashboards show everything is GREEN. Who do you believe — and what do you check first?

Averages hide catastrophic outages. If 900 users complete requests in 20ms and 100 users time out at 30,000ms, the mathematical average is ~3 seconds (which might look 'acceptable' on a coarse dashboard), but 10% of users are completely blocked. Query p95 and p99 latency immediately.

# PromQL query to expose tail latency
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, path))
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, path))

Compare metrics recorded at the client edge (CDN / CloudFront / ALB) against metrics recorded inside the backend application pods. If ALB latency is 5 seconds but application code reports 50ms execution time, the delay is happening in the queuing network layer: TLS handshake overhead, reverse-proxy backlog drops, or slow client transfer.

Green CPU doesn't mean healthy compute. Check for CFS CPU Throttling (`container_cpu_cfs_throttled_periods_total`) where Kubernetes throttles a container due to strict CPU limits even when the node has 80% free CPU. Check database connection pool wait queues and thread lock contention.

# PromQL: Detect hidden CPU throttling on Kubernetes pods
sum(rate(container_cpu_cfs_throttled_periods_total[5m])) by (pod) 
/ 
sum(rate(container_cpu_cfs_periods_total[5m])) by (pod) * 100

Pull up Jaeger/Tempo distributed traces for affected requests. Look at the waterfall visualization: identify whether time is spent waiting on PostgreSQL row locks, DNS resolution, Redis socket connect timeouts, or third-party API gateways.