Label_values (query variable) should support complex inner queries and builder/code editing toggle

Grafana Prometheus
1

  • What Grafana version and what operating system are you using?
    grafana-10.1.5-r0 (open source) on Alpine linux

  • What are you trying to achieve?
    Trying to generate a drop-down variable list derived from label_values query where the metric part is a little more complex than the simplest default assumed and constrained in the wizard.

  • How are you trying to achieve it?

I would like to use a query like this:

label_values(my_metric{a=".*"} * on(a) group_left(a,b) info_metric_a2b{b="$var2"},a)

(of course don’t need to specify a=".", just making it easy to see where it exists for this example)*

  • What happened?
    The query editor when defining dashboard variables doesn’t have the Builder/Code toggle like in dashboard panel editor. So there is no way to specify the raw query. The wizard is too constraining to accept a more complex metric defintion, and efforts to manually type it in are overriden and everything after a space character is dropped.

  • What did you expect to happen?
    I expected to have a query editor option like the great one for defining queries in panel dashboards, with a toggle for Builder and Code.

  • Can you copy/paste the configuration(s) that you are having problems with?
    see query example above.

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

  • Did you follow any online instructions? If so, what is the URL?
    There were no online instructions for this and I can’t find reference of anyone else trying something similar, except this old unanswered link of someone trying to use comparison operators to filter the metric based on metric values to get a subset of label values:
    Full query support when creating variable?

1 Like
2

I want to separately define why this perhaps obscure-seeming request is important and useful. I have many dashboards where I would like to employ the following strategy… in fact this would be my core strategy if it were supported:

  1. Various dashboard variables are used to constrain a large dataset to pick an interesting slice
  2. These dashboard variables are embedded into a potentially complex promql query from which I want to extract label values to generate a list of “interesting items” (e.g. hostnames, or whatever other list of label values).
    3.This list of interesting items captured in the variable field is used as a “repeat by” variable on rows or panels to generate the results

A specific example::
Show graphs of CPU utilization for the top 20 hosts (by CPU utilization) where vendor=“X”,model=“Y”. In this case, the vendor and model labels aren’t directly associated with the metric that contains interface utilization, so it has to be joined with a * on() group_left() info_metric. Further, the metric selection could be wrapped in a promql function such as topk and sum_over_time to achieve the “top 20” part of the functionality. Further, don’t try to hard to exactly constrain to official Prometheus PromQL… I’m using MetricsQL and VictoriaMetrics for better performance and more rich querying capability, and want to be able to specify the raw queries handed to label_values in the same way I can specify raw code for dashboard panel queries to use MetricsQL functions not supported by Prometheus.

In the absence of having this… I’m stuck with “prototype” dashboards leveraging our TERRABYTES of data on 50 MILLION UNIQUE SERIES, showing demos to people about how cool it could be… but I can’t create a production service for our users because they can’t really query by all the interesting fields because we have the labels spread in different info metrics. It would take months to re-write the data based on this Grafana constraint. That would also be a bad strategy because we always discover new metadata that we want to filter by and using “info” metrics is the only practical way to add these rather than re-writing the existing data (which, aside from coding changes, would take weeks each time just to re-write the backfilled data!)

1 Like
3

UPDATE:

I figured out a work-around as follows:

  1. Use VictoriaMetrics datasource plugin
  2. Use a query variable and wrap the arbitrarily complex promql/metricslql in a query_result()
  3. Use the regex field to pick out the label value of interest.

This method has given me the foundation to really build the dashboards I want to build.

1 Like