How to get group by transformation in Grafana dashboard table over joined metrics working correctly?

  • What Grafana version and what operating system are you using?

I am using Grafana 10.0.3 on a K8S cluster.

  • What are you trying to achieve?

I want to create a table of installed software versions based on container images. The table shall show all versions of a software per cluster and if there are multiple instances of that software it should show multiple rows. Also, multiple components should be displayed in one table grouped by topic, e.g. all Elastic versions (elastic, kibana, manager, exporter etc).

  • How are you trying to achieve it?

I have created a table view with N metrics (any other combination of metric leads to the same result, so the metric itself doesn’t matter, I’d assume):

label_replace(
     kube_pod_container_info{container="elasticsearch", image=~".*elastic.*"},
     "elasticversion",
     "$1",
     "image",
     ".*:(.*)"
)

label_replace(
     kube_pod_container_info{container="kibana", image=~".*kibana.*"},
     "kibanaversion",
     "$1",
     "image",
     ".*:(.*)"
)

and 3 transformations:

  1. Filter by name to only include cluster name and version label
  2. Outer join by cluster label
  3. Group-By

  • What happened?

Although there are multiple versions of a software, the table only shows one row:

grafana-groupby-fail

The two top tables show a singular metric w/o any joins. As one can see, there are two versions of kibana and elastic installed on this cluster.

The bottom table shows the rendered result of the above displayed settings: only one row is being displayed.

  • What did you expect to happen?

I expected, that the group-by behaves like the group-by of a database, e.g.:

# select * from test order by cluster;
 cluster | foo | bar  
---------+-----+------
 hanna   | 838 |  119
 hanna   | 838 |  119
 hanna   | 838 |  119
 hanna   |   3 |  119

select * from test group by(cluster,foo,bar) order by cluster;                                                                                                                               
 cluster | foo | bar  
---------+-----+------
 hanna   | 838 |  119
 hanna   |   3 |  119

But instead the dashboard table just “swallows” rows.

I’d appreciate any help.

best,
Tom

PS: I also tried to achieve this using PromQL joins but this failed as well. Also, in some cases I have many more software versions to display. So a join+group-by transformation would be the ideal solution - if it worked.

PPS: or something like this would help as well:

label_replace(
     kube_pod_container_info{container=~"elasticsearch|kibana",cluster=~"$cluster"},
     "$1-version", <--- ! fails
     "$2",
     "image",
     ".*/(.*):(.*)"
)

but is unsupported.

I also tried to just create multiple version labels using label_replace:

label_replace(
    label_replace(
        kube_pod_container_info{container=~"elasticsearch|kibana",cluster=~"$cluster"},
        "elkversion",
        "$2",
        "image",
        ".*/(elasticsearch):(.*)"
    ),
    "kibversion",
    "$2",
    "image",
    ".*/(kibana):(.*)"
)

But when I use this with a table and a group-by transformation, I get this:

That’s better, but now the metrics are not joined by the looks of it.