Q: How do you integrate multiple data sources in Grafana? How do you query metrics, logs, and traces inside the same dashboard panel or correlate them during triage?
Integrate multiple data sources in Grafana using Mixed datasources, transformations, and correlated metric-to-log drill-downs.
Want to master this scenario in a live sandbox? The Linux Foundation's Prometheus Certified Associate (PCA) & Monitoring Labs covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Modern observability stacks combine disparate telemetry tools: Prometheus for numerical metrics, Loki or Elasticsearch for logs, and Tempo for distributed traces. Integrating multiple data sources in Grafana allows engineers to register distinct backends, combine them in a single visualization using the special Mixed data source, and correlate them dynamically using Data Links.
Provision Data Sources via Code
Configure connections declaratively using YAML manifests placed in the provisioning directory rather than manual UI clicks.
# /etc/grafana/provisioning/datasources/datasources.yaml
apiVersion: 1
datasources:
- name: Prometheus-Production
type: prometheus
access: proxy
url: http://prometheus-k8s.monitoring.svc:9090
isDefault: true
jsonData:
timeInterval: "15s"
- name: Loki-Production
type: loki
access: proxy
url: http://loki-gateway.logging.svc:3100- Provisioning Files: Place YAML definitions inside
/etc/grafana/provisioning/datasources/. - Default Flags: Mark one data source as default, set the access mode to
proxy, and inject credentials via environment variables. - Correlation Config: Link trace providers directly to log providers by configuring derived fields inside the data source definition.
Query Multiple Data Sources in Grafana via Mixed Datasource
Combine series from different backends in a single graph panel using the built-in Mixed data source option.
- Select Mixed: In the panel query editor, set the data source dropdown to
-- Mixed --to query multiple data sources in Grafana simultaneously. - Query A: Fetch CPU utilization from Prometheus using PromQL.
- Query B: Fetch system error count from Elasticsearch or CloudWatch using their native query syntax.
- Unified Time Axis: Grafana synchronizes both series onto a shared chronological X-axis, enabling instant visual correlation between metric spikes and log errors.
Correlate with Data Links & Transformations
Connect metrics to logs and traces without manual copy-pasting of timestamps or hostnames.
- Data Links: Configure panel data links that construct URLs pointing directly to Loki or Tempo, automatically passing
${__data.fields.service}and current time boundsfrom=${__from}&to=${__to}. - Transformations: Apply Join by field or Merge transformations to merge tabular data from a SQL database with Kubernetes pod health metrics.
- Provision data sources as code using YAML files inside Grafana provisioning directories.
- Use the built-in Mixed data source to query Prometheus metrics and Loki logs within the same graph panel.
- Configure Data Links on metric panels to open exact log lines or trace IDs in Loki or Tempo preserving timestamp filters.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Configuring all production data sources manually through the Grafana web UI Manual UI configs lead to configuration drift, cannot be peer-reviewed in Git, and break disaster recovery restores.
- Using Direct access mode instead of Server (Proxy) mode for internal data sources Direct mode forces end-user client browsers to establish network connections directly to internal TSDB endpoints, leaking infrastructure topologies.
- Attempting to perform massive SQL joins across two different TSDBs client-side in Grafana Heavy browser-side transformations freeze the client DOM; aggregate data in the source TSDB whenever possible.