Sum of counters with low change rates

Grafana
1

Folks, im completely new to the forum, and most probably going to ask something that has been asked multiple times in the past

it is related to the combination of rate and sum (or even sum_over_time)
see this SO answer

i have the following query for my graph:

  sum by(feature) (
    increase(xyz_service_tokens_histogram_sum{feature="SELDOMLY_USED_FEATURE"}[$__range])
  )

sorry, had to mask some names but the main point is that this query works for most of the features, as they are used heavily, but for this particular one, im getting the graph seen on the screenshot

0_level_sum
0_level_sum1551×289 24.7 KB
.

Why is this a problem? because it shouldn’t be 0 (low number but not 0), when looking at the raw counter metric:

xyz_service_tokens_histogram_sum{feature_scope="SELDOMLY_USED_FEATURE"}

I see what is shown on the second screenshot

individual_counters
individual_counters1569×221 21.3 KB

definitely some non 0 values, but they are not growing at all (most probably because the metric was emitted only once from those envs, it jumped to that level, but as we don’t have datapoints for 0, then those jumps are not evaluated as change)

One thing i forgot to mention: what is my goal?my goal is to get the total token usage over the selected period. i.e. what was the token usage for this feature over the last 90 days

however for this particular case its showing 0 instead of a nonzero value (at least im assuming,t hat it should be non 0)

2

Hi, welcome to the forum.

Unfortunately, the query does exactly what it’s told to do:

  1. It selects every series that match xyz_service_tokens_histogram_sum{feature="SELDOMLY_USED_FEATURE"} (which given your screen are constant)
  2. Calculates the increase on every series (increase of constant function will be 0).
  3. Sums the increases (which are all 0s, so the result is 0).

The main point here is that the increase is applied on all the series one-by-one. What you could do is to firstly calculate the sum, which will give you one time series and then calculate delta on this sum (you shouldn’t use increase(sum(... )

3

thanks for the explanation, it makes sense now