I use lucene query to extract info on how many pods exceeded 20 GB storage threshold over time:
kubernetes.container.rootfs.used.bytes : [20000000000 TO ] AND kubernetes.namespace : AND kubernetes.pod.name: wsg-pw
I use Stat visualization and there i select Distinct Count for calculation:
In the query builder i select Unique Count accordingly:
This works almost fine, except for one thing - the final value displayed in the Stat panel is always deviating by +1 from the actual value. For example, if i see this value displayed:
then i know that there were actually 0 pods that exceeded the storage threshold during the selected time period.
This is because:
- the Distinct Count calculation inside Stat returns number in this fashion:
[0, 1, 2, 3, 4, 5]
Meaning that initially there were 0 pods with exceeded limit, but as the time progressed, then there was 1, then 2, then 3, then 4, then 5 distinct pods that exceeded the limit.
- the Unique Count in the query builder then goes through the array returned by the Distinct Count and finds 6 items in total - including the 0
So i tried to fix it by adding a Math expression like this, but as seen in the printscreen below, it does not work:
So, what am i doing wrong?
Don’t use distinct count in your visualization, when query returns already distinct count.
Use query inspector to debug query output. I bet there will be query result e.g. 5, so count of distinct values in this query result is 1.
@jangaraj you say “dont use distinct count” but but you dont clarify what kind of calculation i should use instead.
If use a simple Count, i get huge numbers which is of course natural because the metrics are sent every 1 minute and thus there are many many datapoints with “kubernetes.container.rootfs.used.bytes” field containing value above 20 GB.
I am already on the right track to extract the distinct values, the only thing is to find out why the applied Math expression doesnt work here
btw, in case you wondering what is the debug output of the query:
You didn’t specify exactly what you need. You have time series so think about time dimension, e.g.
time1 100
time2 1000
Do you you want “current” value for count values → use “Latest” calculation, so result will be 1000
Do you wan to “min” value over time → use “Min” calculation, so result will be 100.
Maybe you have another requirements, …
You case right now:
time1 1443
time2 1443
So number of unique values values (what you configured in the panel calculation) from [1443, 1443] is 1 => that’s exactly what you see - Grafana is doing only what you configured. It is not adding anything.
Recommendations:
- develop query with table panel first (so you are not distracted/confused with any Calculation on the panel level)
- there is no point of returning timeseries, when used panel type doesn’t use timeseries - it should be simple scalar or timeseries with single value only - then any question about calculation on the panel level is irrelevant
It’s pretty clearly stated in my second post - i need to subtract “1” from whatever value is being returned by query “A”.
When i apply “Distinct Count” at the panel level, dont i get the scalar value already?
Take a look at this printscreen:
That value “9” looks pretty scalar to me, so at least in theory i should be able to perform math functions on it, right?
But why cant i do it in practice then?
