Q: Your metrics vendor has an outage. Prometheus remote write queues grow, local disk fills, and the monitoring stack becomes unstable. How do you design for this failure mode?
Remote storage must be treated as a dependency that can fail.
Want to master this scenario in a live sandbox? The Linux Foundation's Prometheus Certified Associate (PCA) & Monitoring Labs covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "In an interview, I explain how we designed actionable, symptom-based alerting using the Four Golden Signals. The interviewer is testing: Remote write backpressure, queue tuning, failure isolation.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Initial Diagnostics & Root Cause Analysis
Remote storage must be treated as a dependency that can fail.
- Tune remote write queue capacity, shard count, retry backoff, and sample age limits.
- Keep local retention sufficient for short vendor outages, but not so large that disks fill silently.
- Alert on remote write failed samples, retried samples, queue length, and WAL disk usage.
- Drop or downsample non-critical metrics during prolonged backend outages.
Remediation & Permanent Safeguards
I would: The goal is graceful degradation: keep critical local alerting alive even when long-term storage is down.
- Run HA Prometheus pairs carefully so both instances do not overload the vendor with duplicate retries.
- Maintain local dashboards for active incidents even if the vendor UI is unavailable.
- Tune remote write queue capacity, shard count, retry backoff, and sample age limits.
- Keep local retention sufficient for short vendor outages, but not so large that disks fill silently.
- Alert on remote write failed samples, retried samples, queue length, and WAL disk usage.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Setting SQS visibility timeout shorter than consumer task processing duration when addressing observability If processing exceeds the visibility timeout, SQS returns the message to the queue, causing concurrent duplicate execution by other workers.
- Omitting a Dead Letter Queue (DLQ) redrive policy for poison pill messages Unparseable poison pill messages fail repeatedly, consuming worker capacity and cycling through retry loops indefinitely without a DLQ.
- Using SNS directly to an HTTP endpoint for mission-critical events requiring guaranteed persistence SNS is an ephemeral push notification service without persistent storage; decoupling via SQS buffers messages during consumer downtime.
- Assuming FIFO SQS queues support unlimited throughput without high-throughput mode enabled Standard FIFO queues cap throughput at 300 messages/sec (3,000 with batching) unless partition-key high-throughput mode is activated.