Time Series of counter rates from Prometheus: organize as (a, b) with an additional label z

Using Grafana 11.2.0

Prometheus metrics are counter(s) with labels “a”, “b”, and “z”. “a” and “b” labels separate counter metrics, but label “z” is solely a useful annotation. I would like to (1) colorize the specific series shown by (a, b) but not z, and (2) still show (a, b, z) as the series display name.

Doing a Prometheus query “sum by (a, b) (rate(counter[15s]))” does the proper colorization but to my limited understanding does not give me access to the z label values in the display names. I’m using the Classic palette here. This is how it looks like:

When I do “sum by (a, b, z) (rate(counter[15s]))” instead, then the colorization with the Classic palette ends up with colors barely distinguishable. That is:

How can I get distinguishable colors and still show the annotation?

The easiest one (though it might be a pain depending on your data) would be using the second query and add overrides for colors (scroll to the lowest point in the visualization settings, there should be Add field override button, there you can pick fields with name and color them the way you like). There probably is a way to add the label within prometheus itself but it’s rather tricky, so I wouldn’t recommend that.

1 Like

I was afraid that the manual overrides would be the only available solution. As we want to use the same dashboard on different system configurations the overrides would more often not match than match. As kind of a workaround I’ve added another table panel that queries the metrics and groups them by (a, b, z), set the query format to “table” and type to “instant” and then transform the data with (1) filter data by values to blank out all the zero elements, and (2) organize fields by name, renaming the fields at the same time. This gives a nice cheat sheet sitting next to the time series panel.