Correlating data sources by environment (Grafana “Experiences”?)

Grafana
1

Imagine an application/service/system that has multiple deployment environments, each with their own Prometheus and Loki instances.

Ideally, in a Grafana dashboard for that application/service/system, I would like to have a single dropdown where I choose an environment from a dynamic (not static) list, and that choice then determines which Prometheus and Loki data sources will be used in the dashboard.

Is there a good way to achieve that today?

While I was looking into this topic, I stumbled upon this blog post from March 2020 briefly mentioning an upcoming grouping mechanism called “Experience”: https://grafana.com/blog/2020/03/31/how-to-successfully-correlate-metrics-logs-and-traces-in-grafana/

Is this still something that is planned to be introduced in Grafana? I tried searching for clues in the Github repo and got a bit overwhelmed by the vast number of results returned when using the search term “Experience”.

2

Hi @jonas4 I was able to achieve something similar for my dashboard with the approach described here in https://github.com/grafana/grafana/issues/28052 where user only select environment name in the drop down and rest of the variables automatically derived based on the first variable.

1 Like
3

Thank you for the suggestion. I played around with the Infinity data source and managed to get a solution working based on which Prometheus and Loki data sources that are actually available.

Grafana Screen Shot

All Prometheus and Loki data sources follow the naming pattern <system>-<env>-<prometheus|loki>, e.g. sys1-env2-prometheus and sys1-env2-loki.

The solution makes use of three variables:

  • DS_PROMETHEUS lists the Prometheus data sources.
  • SYS_ENV is a hidden variable that queries Infinity for Collection($DS_PROMETHEUS). It extracts the <system>-<env> part of the selected Prometheus data source by using the regex /(.*)-prometheus/.
  • DS_LOKI lists the Loki data sources matching the filter /${SYS_ENV}-loki/.
4

I’m still very interested in hearing more about the upcoming grouping mechanism for data sources. Perhaps @aocenas can shed some light on its progress? I would really appreciate that.