Q: How do you troubleshoot missing data in Grafana dashboards ('No data' or empty graphs)? Walk through your systematic diagnostic triage procedure.

Determine if the backend TSDB returned empty results or if Grafana failed to visualize the payload.

• Open Query Inspector: In the panel menu, navigate to Inspect -> Query and click Refresh to inspect backend responses.
• Inspect Raw JSON Response: Review the raw TSDB response. If the array is empty ([]), the database contains no matching data for that time range and query.
• Inspect Error Messages: Look for data source timeout errors (e.g., 504 Gateway Timeout or context deadline exceeded) or invalid syntax errors.

Rule out timestamp offsets and invalid variable interpolation.

# Diagnostic PromQL queries to test target scraping health:
# 1. Check if targets are being scraped successfully:
up{job="payment-service"}

# 2. Check scrape health and errors:
prometheus_target_scrapes_sample_out_of_order_total

# 3. Check when the last metric sample was ingested:
timestamp(up{job="payment-service"})

• Timezone & UTC Offsets: Check if client browser local time is skewed or if the panel is pinned to an absolute past window. Switch dashboard time to UTC.
• Variable Value Check: In Query Inspector, view the compiled query string to confirm variables expanded correctly (e.g. verifying $namespace didn't expand to an empty string or unmatched regex).

Trace backwards to verify that Prometheus is actively scraping and storing metrics from target endpoints.

• Target Scrape Status: In Prometheus web UI or Grafana Explore, execute up{job=""}. If it returns 0, the target is failing health checks or timing out.
• Metric Relabeling & Drops: Inspect Prometheus relabel_configs and metric_relabel_configs to verify series were not unintentionally dropped during scrape ingestion.
• Null Value Display Settings: In the panel Graph styles, check 'Connect null values'. If set to 'Never' and sample intervals vary, graphs display disconnected dots or empty gaps.

Prevent silent data gaps caused by time drift or expired storage blocks.

• NTP Clock Skew: Ensure Grafana, Prometheus pods, and Kubernetes worker nodes synchronize against chrony or AWS Time Sync Service. A 30-second clock skew between Prometheus and Grafana produces false empty windows on real-time dashboards.
• TSDB Retention Horizon: Verify whether historical data falls outside the configured retention window (--storage.tsdb.retention.time=15d). Querying 30 days of data against a 15-day local Prometheus disk returns empty results for the older half of the time window.