Q: Explain the difference between `Gauge` and `Counter` metric types in Prometheus.
Counter: A cumulative metric that can *only go up* (or reset to zero on restart). Examples include http_requests_total or bytes_sent.
Want to master this scenario in a live sandbox? The Linux Foundation's Prometheus Certified Associate (PCA) & Monitoring Labs covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "Logs tell you what happened, metrics tell you where to look, and distributed traces pinpoint the exact slow component. The interviewer is testing: Fundamental metric types.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Initial Diagnostics & Root Cause Analysis
- Counter: A cumulative metric that can *only go up* (or reset to zero on restart). Examples include
http_requests_totalorbytes_sent. Because it only goes up, you never query its raw value directly; you always apply a rate function (e.g.,rate(http_requests_total[5m])) to see how fast it's growing. - Gauge: A metric that can arbitrarily *go up and down* over time. Examples include
cpu_memory_usage,current_queue_depth, ortemperature. You can query gauges directly to evaluate their current value without needing a rate function.
Remediation & Permanent Safeguards
Execute the resolution runbook and verify workload health:
- Counter: A cumulative metric that can *only go up* (or reset to zero on restart). Examples includ...
- Gauge: A metric that can arbitrarily *go up and down* over time. Examples include cpu_memory_usag...
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Injecting high-cardinality label dimensions (such as user IDs or request UUIDs) into Prometheus metrics when configuring Observability Explain difference High-cardinality label values cause exponential time-series growth, consuming gigabytes of TSDB memory and crashing Prometheus servers.
- Evaluating application performance solely through arithmetic mean or p50 latency while ignoring tail latency (p99) when configuring Observability Explain difference Averages hide catastrophic tail latency outages where 1% of high-value user transactions suffer severe timeouts.
- Failing to inject and extract W3C distributed trace context across message brokers and RPC calls when configuring Observability Explain difference Breaking distributed trace context propagation fragments end-to-end transaction waterfalls into disconnected, un-correlatable trace segments.
- Configuring alert rule evaluation intervals without sufficient duration damping (for: 5m), inducing alert flap when configuring Observability Explain difference Instant alerts triggered by transient sub-second CPU or latency blips page on-call engineers unnecessarily and dilute response urgency.