Dashboard variable for getting all values from a label is not working in UI

[zakirefai]

• What Grafana version and what operating system are you using?

  • We are using self managed v10.4.3 (6e6fbf6a34)

• What are you trying to achieve?

  • We are trying to add a variable on our dashboard that will return all possible values for the label testid. With this variable, we will be able to select All, a single testid, or multiple testids. The testid label exists in our k6 metrics, which are pushed to our Prometheus DB, and can look like this {__name__="k6_vus", scenario="load", testid="<testid>"}

• How are you trying to achieve it?

  • We would like to be able to easily set this up through the UI.

• What happened?

  • Every query we have tried returns empty results or a Bad request error.

• What did you expect to happen?

  • The query is able to return the list of testids found across all k6 metrics.

We have been trying to figure this out for days now, and unable to find a reliable solution. When we attempt to create the variable described above, we always run into errors or empty results. Going through the UI normally, here is what we have tried.

Using the default k6 Grafana dashboard available for download
When we pulled down the available k6 Grafana dashboard (dashboard here) and tried to play with the Test Id variable, we always get this error.

This led me to believe that not all metrics have the testid variable, so I tried to drill down to a specific metric.

Changing the metric to k6_vus, I can’t even drill down into the labels. You can see this in the picture below, as the options are only the dashboard variables and not the labels within the metric.

This is concerning to me because I am able to query this same metric in the explorer and get back the labels as seen below; like scenario and testid.

The testid label exists in all of the k6 metrics posted to the DB.

The dashboard is just unable to find them through the UI.

I have tried changing the query type in the UI. I have tried several different queries under each type. I always either get an error or an empty result for the queries. Here is a small list of queries I’ve tried under each query type:

• Query result:
  • group by (testid) (k6_vus_max)
    • Ran this one in explorer and got back each individual testid. It seemed like this query would work, but plugging it into the dashboard yielded an empty result.
• Series query:
  • k6_vus{testid="[a-zA-Z]"}
    • Tried to use regex to select any value in the testid label. Got an empty result back for this one.
• Classic query:
  • label_values(k6_vus, testid)
    • Returns an empty result. This one is odd since it doesnt throw an error. This exact query will throw an error if I use the Label values query instead.

Any help here would be outstanding. Is there anything I can do here or take a look at that might help me drill into the problem? I am genuinley stuck, as it seems like Grafana is fighting me on this.

[geromari1008]

have same issue in 11.2.0 . it does not find any vlaue at all or it will list all values of the label by ignoring other label filters.
another odd thing is that it is working well with one datasource and not with other but all of them are same type prometheus , just 3 different clusters.

workaround for that problem is for me to use group by + regexp /.id="((.)?)"./

@zakirefai

• What Grafana version and what operating system are you using?

  • We are using self managed v10.4.3 (6e6fbf6a34)

• What are you trying to achieve?

  • We are trying to add a variable on our dashboard that will return all possible values for the label testid. With this variable, we will be able to select All, a single testid, or multiple testids. The testid label exists in our k6 metrics, which are pushed to our Prometheus DB, and can look like this {__name__="k6_vus", scenario="load", testid="<testid>"}

• How are you trying to achieve it?

  • We would like to be able to easily set this up through the UI.

• What happened?

  • Every query we have tried returns empty results or a Bad request error.

• What did you expect to happen?

  • The query is able to return the list of testids found across all k6 metrics.

We have been trying to figure this out for days now, and unable to find a reliable solution. When we attempt to create the variable described above, we always run into errors or empty results. Going through the UI normally, here is what we have tried.

Using the default k6 Grafana dashboard available for download
When we pulled down the available k6 Grafana dashboard (dashboard here) and tried to play with the Test Id variable, we always get this error.

This led me to believe that not all metrics have the testid variable, so I tried to drill down to a specific metric.

Changing the metric to k6_vus, I can’t even drill down into the labels. You can see this in the picture below, as the options are only the dashboard variables and not the labels within the metric.

This is concerning to me because I am able to query this same metric in the explorer and get back the labels as seen below; like scenario and testid.

The testid label exists in all of the k6 metrics posted to the DB.

The dashboard is just unable to find them through the UI.

I have tried changing the query type in the UI. I have tried several different queries under each type. I always either get an error or an empty result for the queries. Here is a small list of queries I’ve tried under each query type:

• Query result:
  • group by (testid) (k6_vus_max)
    • Ran this one in explorer and got back each individual testid. It seemed like this query would work, but plugging it into the dashboard yielded an empty result.
• Series query:
  • k6_vus{testid="[a-zA-Z]"}
    • Tried to use regex to select any value in the testid label. Got an empty result back for this one.
• Classic query:
  • label_values(k6_vus, testid)
    • Returns an empty result. This one is odd since it doesnt throw an error. This exact query will throw an error if I use the Label values query instead.

Any help here would be outstanding. Is there anything I can do here or take a look at that might help me drill into the problem? I am genuinley stuck, as it seems like Grafana is fighting me on this.