Using the "All" variable option when using unique data sets

Grafana Prometheus
1

I am trying to build a dashboard using 2 sets of data: one set has the server names as FQDN (k8s related) and the other is just the simple server name (node exporter data). I want to show some metrics for the nodes from the node_exporter data.

My Problem:
If I have my “node” variable set to allow “ALL” as an option (so I have metrics from all nodes) then the node_exporter queries also use the “all” which returns metrics for servers outside the cluster.
My variable query:

image
image719×147 7.32 KB

which works great but then I either need to make sure and reselect all the nodes when I start the dashboard or change the $datasource OR if I use the all I get the issue of all servers being returned for my node_exporter query. There are not any other common variables I can use to filter out the node_exporter metrics. I tried to have a hidden variable for the node_exporter data but was unable to get it to auto select all values that came in from the main variable without it also processing ALL even if the hidden variable does not allow an all value.
image
image1053×505 37.7 KB

Possible solutions:

  • Is there a way to process a variable’s actual values without the “all” getting passed on? This could include forcing a variable to always select all values in the drop box without having all
1 Like
2

Hi,

It’s simple to do it…
Just remove the key that contain variable $node in the query, then you can have all the values.

example:
some_metrics{node="$node", datasource="$datasource"}
to
some_metrics{datasource="$datasource"}

Then you will gel all values without consider the node.

Regards,
Fadjar Tandabawana

3

Thanks @fadjar340 for the response but the variable $node is the common point I need to select my node_exporter metrics so they align with the nodes of our k8s cluster.
I thought about trying to do a join but I have the issue that I don’t have a direct label alignment (both in name and one being fqdn and one not) and I have almost a years worth of data and almost 15-25 Prometheus instances I would have to refactor depending how I changed it.

4

So Ended up being self inflicted - in the “custom all value” There was an * and that was causing the returned list to show all VMs rather than the filtered list from the first variable.

1 Like