Q: A developer comes to you saying they cannot find an error log in Datadog/Kibana that they just triggered in production. You verify the application is generating the log. Why is it missing?
Logs do not magically appear in aggregators; they traverse a pipeline. I would check:
Want to master this scenario in a live sandbox? The Linux Foundation's Prometheus Certified Associate (PCA) & Monitoring Labs covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "Our SRE team tackled this monitoring and metrics bottleneck to eliminate false-positive alert fatigue. The interviewer is testing: Log shipping path, ingestion latency, parsing filters.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Initial Diagnostics & Root Cause Analysis
Logs do not magically appear in aggregators; they traverse a pipeline. I would check:
- Ingestion Latency: There might simply be a delay in processing logs if logstash/fluentd is backlogged. Check the lag metrics on the log shipper.
- Quota/Rate Limiting: The log aggregator (like Datadog/Splunk) might be silently dropping logs because the daily index/ingestion quota was breached.
- Parse Failures (Grok patterns): If the developer changed the log format in the latest deployment, the log shipper might fail to parse the JSON or regex pattern, sending it to a dead-letter queue or dropping it.
Remediation & Permanent Safeguards
- Log level: Ensure the production environment is actually configured to output
DEBUG/INFO(often it's set toWARN/ERRORonly).
- Ingestion Latency: There might simply be a delay in processing logs if logstash/fluentd is backlo...
- Quota/Rate Limiting: The log aggregator (like Datadog/Splunk) might be silently dropping logs bec...
- Parse Failures (Grok patterns): If the developer changed the log format in the latest deployment,...
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Injecting high-cardinality label dimensions (such as user IDs or request UUIDs) into Prometheus metrics when configuring Observability developer comes High-cardinality label values cause exponential time-series growth, consuming gigabytes of TSDB memory and crashing Prometheus servers.
- Evaluating application performance solely through arithmetic mean or p50 latency while ignoring tail latency (p99) when configuring Observability developer comes Averages hide catastrophic tail latency outages where 1% of high-value user transactions suffer severe timeouts.
- Failing to inject and extract W3C distributed trace context across message brokers and RPC calls when configuring Observability developer comes Breaking distributed trace context propagation fragments end-to-end transaction waterfalls into disconnected, un-correlatable trace segments.
- Configuring alert rule evaluation intervals without sufficient duration damping (for: 5m), inducing alert flap when configuring Observability developer comes Instant alerts triggered by transient sub-second CPU or latency blips page on-call engineers unnecessarily and dilute response urgency.