Using absent with templated vars

I am having trouble creating what I would think is a trivial dashboard panel.
I want a list of node exporters whose data is not making it to grafana.
How can I achieve this? I have a dashboard with a custom variable that lists all of the racks.
If I query up {job=“node”, rack=~“$rack”} I get a time series of the node exporters that are up, but any down node exporter is just left out of the graphs.
If I query absent( up{job=“node”, rack=~“$rack”} ) - this works if 1 missing item is chosen for the rack variable, but if multiple are choosen and any one of those is up, then nothing is returned.
What is a good way to achieve this? I feel I must be missing something obvious.

  • What Grafana version and what operating system are you using?
    10 Linux

  • What are you trying to achieve?
    Time Series graph of node exporters that are not available/not exporting.

  • How are you trying to achieve it?
    absent( up{job=“node”, rack=~“a|b|c”} )

  • What happened?
    The result is not calculated for each rack ie I do not get a list of racks that are down
    If a or b or c is up, then return value is No data

  • What did you expect to happen?
    Each rack that is down would be included in the returned data.

I would say that absent is working exactly as it is documented:

returns an empty vector if the vector passed to it has any elements (floats or native histograms) and a 1-element vector with the value 1 if the vector passed to it has no elements.

Thanks for clarifying that. Seems obvious now :person_facepalming:
Is there a way to create a query to find missing “up” jobs from a list?

See 5. Prometheus monitoring with labels part missing:

Idea is to compare label values, which were in the prometheus metric in the past with current label values. I would say this better option + you need to include also templated vars into your query.