Q: You are building an alerting strategy for a newly launched microservice. What are the four 'Golden Signals' you should base your SLIs on?
Google's SRE book defines four "Golden Signals" as the baseline for user-facing systems:
Want to master this scenario in a live sandbox? The Linux Foundation's Prometheus Certified Associate (PCA) & Monitoring Labs covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "Our SRE team tackled this monitoring and metrics bottleneck to eliminate false-positive alert fatigue. The interviewer is testing: Google SRE best practices, Golden Signals.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Initial Diagnostics & Root Cause Analysis
Google's SRE book defines four "Golden Signals" as the baseline for user-facing systems:
- Latency: The time it takes to service a request (differentiating between successful and failed requests).
- Traffic: A measure of how much demand is being placed on your system (e.g., HTTP requests per second).
- Errors: The rate of requests that fail (e.g., explicitly HTTP 500s or implicitly corrupt data).
Remediation & Permanent Safeguards
- Saturation: How "full" your service is. A measure of the most constrained resource (e.g., CPU, Memory, I/O, or database connection pool utilization).
- Latency: The time it takes to service a request (differentiating between successful and failed re...
- Traffic: A measure of how much demand is being placed on your system (e.g., HTTP requests per sec...
- Errors: The rate of requests that fail (e.g., explicitly HTTP 500s or implicitly corrupt data).
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Injecting high-cardinality label dimensions (such as user IDs or request UUIDs) into Prometheus metrics when configuring Observability building alerting High-cardinality label values cause exponential time-series growth, consuming gigabytes of TSDB memory and crashing Prometheus servers.
- Evaluating application performance solely through arithmetic mean or p50 latency while ignoring tail latency (p99) when configuring Observability building alerting Averages hide catastrophic tail latency outages where 1% of high-value user transactions suffer severe timeouts.
- Failing to inject and extract W3C distributed trace context across message brokers and RPC calls when configuring Observability building alerting Breaking distributed trace context propagation fragments end-to-end transaction waterfalls into disconnected, un-correlatable trace segments.
- Configuring alert rule evaluation intervals without sufficient duration damping (for: 5m), inducing alert flap when configuring Observability building alerting Instant alerts triggered by transient sub-second CPU or latency blips page on-call engineers unnecessarily and dilute response urgency.