Q: How do you integrate Grafana with Prometheus and the ELK stack? How do you correlate Prometheus metric spikes directly with Elasticsearch logs during an incident?
Integrate Grafana with Prometheus and ELK to correlate numerical metrics with Elasticsearch log streams via unified dashboards and Data Links.
Want to master this scenario in a live sandbox? The Linux Foundation's Prometheus Certified Associate (PCA) & Monitoring Labs covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: When you integrate Grafana with Prometheus and ELK, you create a unified observability platform that correlates numerical metrics with rich log events. Rather than forcing engineers to jump between Grafana and Kibana during outages, linking both backends provides a single pane of glass where Prometheus metric spikes link directly to Elasticsearch log records.
Configure Prometheus Data Source
Establish high-performance metric scraping connectivity.
- Endpoint & Proxy: Connect to Prometheus via its internal Kubernetes service (e.g.
http://prometheus-server:9090) using Server access mode. - Scrape Interval Alignment: Set Scrape interval in data source settings to match your Prometheus server scrape target interval (typically
15sor30s). - HTTP Method: Set HTTP method to
POSTfor long-running queries to avoid URL length limitations with large regex matchers.
Configure Elasticsearch / OpenSearch Data Source
Connect Grafana directly to your Elasticsearch or OpenSearch cluster indices.
# Provisioning configuration for Prometheus + Elasticsearch:
apiVersion: 1
datasources:
- name: Prometheus-Prod
type: prometheus
url: http://prometheus:9090
access: proxy
isDefault: true
- name: Elasticsearch-Logs
type: elasticsearch
url: https://elasticsearch.logging.svc:9200
access: proxy
database: "filebeat-*"
jsonData:
timeField: "@timestamp"
esVersion: "8.0.0"
logMessageField: "message"
logLevelField: "log.level"- Index Pattern: Specify the time-based index pattern, such as
filebeat-*,logstash-*, orapp-logs-*. - Time Field & Version: Define the primary timestamp field (typically
@timestamp) and specify the exact Elasticsearch version (e.g.8.x+) for query syntax alignment. - Logs Panel Rendering: Use Grafana's dedicated Logs panel visualization to render log streams with color-coded log levels (ERROR, WARN, INFO).
Integrate Grafana with Prometheus and ELK via Data Links
Build seamless transitions from Prometheus latency spikes to Elasticsearch error logs.
- Configure Data Link: In the Prometheus metric graph panel, add a Data Link pointing to the Logs dashboard or Explore view.
- Parameter Passing: Pass the selected service name and the exact incident time window:
/explore?left=["${__from}","${__to}","Elasticsearch-Logs",{"query":"service.name:${__series.name} AND log.level:error"}]. - One-Click RCA: When an engineer clicks an anomalous latency spike on the graph, Grafana instantly opens the exact matching Elasticsearch error logs for that specific minute.
- Connect Prometheus via HTTP POST for metrics, and Elasticsearch via index patterns like filebeat-* for logs.
- Use Grafana's native Logs panel to visualize structured logs alongside CPU and latency graphs.
- Add Data Links to Prometheus metric panels that pass timestamp bounds and service names directly into Elasticsearch log queries.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Running wildcard Lucene queries without index restrictions against multi-terabyte Elasticsearch clusters Unfiltered Elasticsearch queries scan months of cold shards, exhausting heap memory and triggering query timeouts.
- Forcing engineers to use Kibana for logs and Grafana for metrics without unified cross-linking Engineers waste critical minutes manually copying timestamps and pod names between tools during production incidents.
- Setting Elasticsearch data source scrape interval or time field incorrectly Mismatched time fields cause logs to appear out of order or fail to load entirely inside Grafana panels.