PromQL won't handle null values during vector matching


I have two PromQL queries in Grafana.

Query 1: max_over_time(counter{label="label1"}[5m])
Query 2: max_over_time(counter{label="label1"}[5m] offset 10m)

There’s an exact match between the labels in both queries, so I don’t believe I need to use the on() function.
I would like to compute the difference between these queries…

Query 3: max_over_time(counter{label="label1"}[5m]) - max_over_time(counter{label="label1"}[5m] offset 10m)

Query 3 returns a resulting vector which is correct for the most part. If the resulting vector of Query 1 has an entry, at the i’th position, with value 1500 and the resulting vector of Query 2 has an entry at the i’th position with value 1000. Then the i’th position of the resulting vector in Query 3 becomes 500.

But when Query 1 has a value of 1000 and Query 2 a value of null (which is formatted as 0), the result becomes 1000 - null = 0.

I would obviously like the result to be 1000 in this case. I have attempted to convert all null values to zero, but based on what I’ve read, Prometheus seems to already treat nulls as zeros. I have also attempted to use vector(0):

max_over_time(counter{label="label1"}[5m]) or vector(0) - max_over_time(counter{label="label1"}[5m] offset 10m) or vector(0)

But this doesn’t change the result.

I would really appreciate some helpful tips as I’ve just started learning PromQL. Thanks in advance.


Expected: (..., 1500, 1000, ...) - (..., 1000, null, ...) = (..., 500, 1000, ...)

Actual: (..., 1500, 1000, ...) - (..., 1000, null, ...) = (..., 500, 0 ...)

Have you tried something like increase(max_over_time(counter{label="label1"}[5m])[10m:])?

Doesn’t always do the job in every case, but could be enough to solve your problem here.