I want Grafana dashboard to show data only from all targets of one prometheus job

Hello. I have a very annoying problem with configuration of the Grafana dashboard.

I have Prometheus that has two jobs configured. And each of those jobs has multiple targets. And I want to have a single dashboard that will allow me to select either I want data from one job or the another job. Both jobs’s targets aim to the same type of applications (cAdvisor, NodeExporter).

I know that in the dashboard’s variables section I can input into Query something like this: label_values(job) and it returns exacly what I want - two different job names from prometheus configuration… image
BUT
I cannot query the data using those values! I tried:
($server is set to either the first job name or the second job name)

((node_memory_MemTotal_bytes{job=~"$server:."} - node_memory_MemAvailable_bytes{job=~"$server:."}) / node_memory_MemTotal_bytes{job=~"$server:.*"}) * 100

And it returns nothing… What actually works is when i change the “job” to “instance” and change the prometheus variable query to label_values(instance) But this query returns the targets themselves. So i get cAdvisor address with port twice (from both jobs) and node exporter with port twice (also from both jobs) and I have to select multiple instances on the dashboard in the Node selection to get the data: I do not want it this way because it is a lot of mess… image

Please help me with this… How to get data on grafana dashboard from all targets from ONE JOB ONLY for all the panels?

1 Like