SOLVED Display Counter Increase in Grafana

I had this problem where in grafana I wanted to show the increase of a gauge or counter over time. I didn’t want to show the absolute value, but the increase.

(I wanted to embed more pictures, but couldn’t because I can only embed 5 in my first post. Head to my blog to see it in full.)

There is no builtin function to handle showing increases where a time serie possibly has null values and drops, like for example after a deploy where it starts from 0. So here are the steps to achieve this.

Say you have this time series:

and you want to produce this:

We go from 7 to 8 at the start, then go from 0 to 5 at the end, that makes a total increase of 6.

First, you need to duplicate the time series with an offset of $__interval, here I plot both without (A) and with offset (B):

A:  sum (my_gauge)
B:  sum (my_gauge offset $__interval)

A naive A-B operation does not produce a good result: the total is wrong and we can see negative values

A:  sum (my_gauge)
B:  sum (my_gauge offset $__interval)
C:  $A - $B

let’s first get rid of negative values:

C:  ($A - $B) >= 0

The total is better, but it’s not 6 just yet.

You can see where the vertical bar is, we do not register the jump from 0 to 1, because there is no data point. To add data points, we need to do this:

A:  sum (my_gauge) or vector (0)
B:  sum (my_gauge offset $__interval) or vector (0)

If we again print the difference, we can see the total is accurate:

C:  ($A - $B) >= 0

(see blog for pic, sry, couldn’t post more than 2 links…)

But there’s a not pleasing gap there when the values drop. The final calculation is:

C:  ($A - $B) >= 0 or vector (0)

The actual calculation I have is this beautiful horrendous formula:

    (sum (my_gauge)                    or vector (0))
  - (sum (my_gauge offset $__interval) or vector (0))
) >= 0 or vector (0)

which gives the result.

1 Like