Grafana no data points displayed when variable filter set to “All”

Grafana Prometheus
1

I used variable – “cluster” : label_values( up{job="$job"}, cluster )
I clicked on “ALL” in this cluster variable – however no data shows up.
I wrote bunch of more variables-ex-- “tenant” : label_values(metric{job="$job", cluster=~"$cluster"}, user) and it can filter correctly for ALL tenants in different cluster.

But this query does not work for selecting ALL cluster and ALL tenants. How do I get data to select ALL cluster? Am I supposed to add another variable in the list of variables? I am just very confused and unable to understand what to do.

Can someone please help-- I have already spent 3 days trying to figure this and this is due today…

2

Try this:

cluster : label_values(up{job='$job'},cluster)

Check the result using Update button
For tenant:

tenant: lael_values(metric{job='$job',cluster=~'$cluster'},user)

Dont use “$job” just ‘$job’

3

I have tried that and it does not work.

Yes I am able to select ALL for tenant and display result.

But I am unable to display ALL for cluster i.e. label_values(up{job=“job”}, cluster) does not show results for ALL when I toggle ALL label for cluster as well. I am not sure how to display results when I select ALL.

4

Use label_values(up{job=~"$job"},cluster). Sign ~ was missing. It triggers regex matching, which is what Grafana produces when you select All. It’s all values concatenated with | sign acting as “or”.