TLDR; Dashboard variable seems to be extrapolating wrong when using multi-value in dashboard settings. Reason: my lack of experience or bug.
Hello,
I’ve went through the documentation, watched tutorials on YouTube, researched blogs and this community forum, but my issue is pretty specific and nothing has popped up. Keep in mind I’ve been using Grafana for just a few months, so I may have missed something really obvious.
- What Grafana version and what operating system are you using?
8.4.6 oss on k8s (not sure the underlying distro)
- What are you trying to achieve?
My job is to create a dashboard that has two panels: with apache metrics and mysql metrics.
- How are you trying to achieve it?
I have two VMs:
vm_1
role: apache
exporter: apache_exporter
IP: a.b.c.d
vm_2
role: mysql
exporter: mysqld_exporter
IP: e.f.g.h
I see metrics from them in Grafana. And their respective library panels are working in separation (when in separate dashboards like mysql-all or apache-all).
Note: both apache and mysql panels use instance="$instance" inside their respective PromQL queries.
I created a dashboard with instance as a query variable:
up{job=~"REDACTED-apache|mysqld-exporter", service!="mon-prometheus-node-exporter"}
Note: I’m removing k8s metrics mon-prometheus because I don’t need them in this dashboard.
- What happened?
All panels are empty:
When using a normal query variable (non multi-value) I see either the apache or mysql library panels populate (depending on what I pick from the drop-down).
- What did you expect to happen?
I wanted to be able to use the check-boxes provided by the “multi-value” variable.
- Can you copy/paste the configuration(s) that you are having problems with?
apache panel
apache_cpuload{instance="$instance"}
mysql panel
rate(mysql_global_status_innodb_data_read{instance="$instance"}[1m])
rate(mysql_global_status_innodb_data_written{instance="$instance"}[1m])
rate(mysql_global_status_innodb_data_fsyncs{instance="$instance"}[1m])
- Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
When using the query inspector, I see a difference between the actual executed queries.
NOTE the \\\\ in the second example
The same library panel: apache cpu load:
in the apache2 dashboard it works:
Object
request:Object
url:"api/ds/query"
method:"POST"
data:Object
queries:Array[1]
0:Object
datasource:Object
uid:"prometheus"
type:"prometheus"
exemplar:true
expr:"apache_cpuload{instance=\"172.16.16.10:9117\"}"
in the my-testing dashboard the same library panel does not work:
Object
request:Object
url:"api/ds/query"
method:"POST"
data:Object
queries:Array[1]
0:Object
datasource:Object
uid:"prometheus"
type:"prometheus"
exemplar:true
expr:"apache_cpuload{instance=\"172\\\\.16\\\\.16\\\\.10:9117\"}"
I’m completely open to the idea that I’m doing something dumb. How can I make this work?
- Did you follow any online instructions? If so, what is the URL?
Many of my previous searches fall into the category of close, but not close enough:
https://grafana.com/docs/grafana/latest/variables/formatting-multi-value-variables/
https://community.grafana.com/t/query-with-variable-does-not-return-data-when-multi-value-is-enabled/37879
https://grafana.com/docs/grafana/latest/variables/advanced-variable-format-options/#raw
https://www.reddit.com/r/grafana/comments/pprmfp/repeating_query_per_multivalue_variable/
https://github.com/grafana/grafana/issues/7205
https://community.grafana.com/t/single-graph-with-multiple-plots-for-multi-value-template-variable/856/6
https://github.com/grafana/grafana/pull/11828#issuecomment-441205019
https://github.com/JamesOsgood/mongodb-grafana/issues/31
https://grafana.com/docs/grafana/latest/variables/filter-variables-with-regex/
https://sbcode.net/grafana/dashboard-variables/
https://dzone.com/articles/how-to-use-grafana-variables-to-make-more-interact
