Q: An application generates 500,000 log lines per second. Storing everything costs $100,000/month. You need detailed debugging capability but cannot afford full-volume storage. What sampling strategy allows you to capture errors while discarding routine logs?
Log sampling reduces volume while preserving critical signals. There are two approaches:
Want to master this scenario in a live sandbox? The Linux Foundation's Prometheus Certified Associate (PCA) & Monitoring Labs covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "During a high-traffic production event, our observability stack proved essential in isolating this latency surge. The interviewer is testing: Log cost optimization, sampling strategies, tail-based sampling.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Initial Diagnostics & Root Cause Analysis
Log sampling reduces volume while preserving critical signals. There are two approaches:
- All logs from *error* requests (even if they're low-volume)
- All logs from *slow* requests (latency > 1000ms)
- Random sample of *success* requests (1% to track healthy profiles)
- Full volume: 500,000 logs/sec × $10 per million logs = $150,000/month
- With tail-based sampling (errors + 1% sample):
Remediation & Permanent Safeguards
1. Head-Based Sampling (Probabilistic): At the point where the log is generated, randomly decide: keep this log with probability P (e.g., 1% of logs). The decision is made instantly, lowest CPU overhead. Problem: You randomly discard errors. With 1% sampling, you'll miss 99% of the stack traces. 2. Tail-Based Sampling (Intelligent): Capture *all* logs in a temporary buffer, but only ship to the aggregator if they match certain criteria: A log forwarder like Fluentbit or OpenTelemetry Collector buffers logs in memory as they arrive, tags them with request outcome (error/success/latency), and makes shipping decisions *after* the request completes. Example (with OpenTelemetry): Cost Math:
import random
if random.random() < 0.01: # Keep 1%
logger.info("request completed")
- Errors: ~5,000/sec, success sample: ~5,000/sec = 10,000 logs/sec
- Monthly cost: 10,000 × 86400 × 30 × $10 / 1,000,000 = $2,592/month (98% savings)
- Debugging capability: All errors captured + representative success traces for normal behavior analysis.
- All logs from *error* requests (even if they're low-volume)
- All logs from *slow* requests (latency > 1000ms)
- Random sample of *success* requests (1% to track healthy profiles)
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Injecting high-cardinality label dimensions (such as user IDs or request UUIDs) into Prometheus metrics when configuring Observability application generates High-cardinality label values cause exponential time-series growth, consuming gigabytes of TSDB memory and crashing Prometheus servers.
- Evaluating application performance solely through arithmetic mean or p50 latency while ignoring tail latency (p99) when configuring Observability application generates Averages hide catastrophic tail latency outages where 1% of high-value user transactions suffer severe timeouts.
- Failing to inject and extract W3C distributed trace context across message brokers and RPC calls when configuring Observability application generates Breaking distributed trace context propagation fragments end-to-end transaction waterfalls into disconnected, un-correlatable trace segments.
- Configuring alert rule evaluation intervals without sufficient duration damping (for: 5m), inducing alert flap when configuring Observability application generates Instant alerts triggered by transient sub-second CPU or latency blips page on-call engineers unnecessarily and dilute response urgency.