Hello. I have a kubernetes cluster with Grafana v8.3.4 backed by Prometheus backed by Postgresql. I have a counter stat that is reported from multiple nodes in the k8s cluster. I have a stat panel that shows the total increase in the counter across all of the nodes using a delta calculation. It works perfectly unless there has be a counter reset or a gap in the data from one of the nodes. I can fix it for either the gap or counter reset by using a different calculation, but not both.
It seems like this would be a common problem but I haven’t been able to find a solution for it short of writing a custom plugin to deal with it.
The query for the stat panel is:
sum(data_count{name=“data_inserted”})
Using the particular time range I’m looking at that contains both a gap in data from one of the nodes and a counter reset later on that same node the value that it comes back with using the Delta calculation is:
94,044,044
If I remove the sum, using data_count{name=“data_inserted”}, and use the same Delta then the three nodes come back with:
17,241
23,207
25,015
What I would expect is that the sum of those three values (which are correct) is what I should be getting with the sum above (that comes back with 94m).
Looking at the datapoints from the query inspector with the individual node stats shows:
“data”: {
“values”: [
[
1651797780000,
1651797840000,
1651797900000,
1651797960000,
1651798020000,
1651798080000
],
[
34552753,
34564334,
null,
34569979,
34569979,
15
]
]
}
“data”: {
“values”: [
[
1651797780000,
1651797840000,
1651797900000,
1651797960000,
1651798020000,
1651798080000
],
[
15955224,
15967253,
15978431,
15978431,
15978431,
15978431
]
]
}
“data”: {
“values”: [
[
1651797780000,
1651797840000,
1651797900000,
1651797960000,
1651798020000,
1651798080000
],
[
13716194,
13727354,
13740784,
13741209,
13741209,
13741209
]
]
}
So in the first set of points you can see the gap and the counter reset. Now if I look at the data points from the sum this is what I see:
“data”: {
“values”: [
[
1651797780000,
1651797840000,
1651797900000,
1651797960000,
1651798020000,
1651798080000
],
[
64224171,
64258941,
29719215,
64289619,
64289619,
29719655
]
]
}
With that data using Delta the result it comes back with is the 94m I listed above. If I change to range (which will obviously be wrong given the data points that are returned above) it comes back with 34.5m and if I change to difference then it comes back with -34.5m.
Is there some way to deal with this? to get the answer I’m looking for here? I would expect to get the sum of the three individual node stats, which would be 17,241 + 23,207 + 25,015 = 65,463.
Thanks!