Time series graph with % of total count for timewindow instead of absolute count

Grafana Prometheus
1
  • What Grafana version and what operating system are you using?

Grafana v11.1.0 (Open source)

  • What are you trying to achieve?

I am trying to create time series graph. Normally we get count of metrics for every time-window. However, I want to plot % of total for every time window (i.e. every 5 mins or 15 mins,etc.). How do I do it?

  • How are you trying to achieve it?

NA

  • What happened?

I tried but no success. I am looking for help on promql/Grafana query.

  • What did you expect to happen?

I am expecting time series graph with % of total count for every time-window instead of absolute count.

  • Can you copy/paste the configuration(s) that you are having problems with?

NA

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

No

  • Did you follow any online instructions? If so, what is the URL?

No

2

Create dashboard variable which will count that “total” - no idea if it is possible on the dashboard variable level, so test/develop that.
Use that variable value in your panel PromQL query to calculate percentage for each time bucket.

3

Thanks but I couldn’t crack this yet. Nevertheless, let’s close this thread.

4

Considering your metric (OpenMetrics format) is like:

# counter
requests{status="500"} 10
requests{status="200"} 20
requests{status="503"} 5
requests{status="201"} 30
...

What would work (I think) is some query like:
sum(rate(requests[5m])) by (status) / on() group_left sum(rate(requests[5m]))

Where 5m is the desired time window. What it would do:
requests[5m] takes every point the metric reported in that time window
rate(requests[5m]) makes (< last point value > - < first point value >) / 240s (time in between those points) - look at the image (last point = 241, first point = 1, time in between those point - 240s):

image
image1308×357 25 KB

Then sum(rate(requests[5m])) calculates the sum of all the series.

For the fun part - on the left side of the / operator we group the requests by statuses. The on() group_left part is because of Prometheus vector matching - in short - when prometheus divides (or does any arithtmetic operation) between two vectors (this case), it looks at the labelsets on both sides and tries to match identical sets. Since we have {status="<status>"} labelset on the left side and {} (empty) labelset on the right side, prometheus cannot match labels, therefore returning no data. That’s the on() part - we tell prometheus to match on empty set of labels. Since there are multiple statuses on the left side, we have many-to-one matching, so Prometheus would go crazy that you cannot do that, since then you’d have multiple series with the same labelset (we ignored the statuses), which is illegal. We need to tell prometheus “group by the labels from the left side” with the group_left part.

I really hope that was what you were looking for and all of this response wasn’t in vain :smile: