Really frustrated about simple counting of particular events during selected time.
I’ve got a simple dashboard, where I have log lines from LOKI (on the screenshot - filtered by “ERROR” keyword), some graphs, etc.
So… as we can see on the 1st screenshot Errors count in the logs really differs from the “calculated”.
What this " 1.9 " errors mean?
1.9 (even 19) in selected by user 5 minutes period? Wrong!
1.9 per second?.. Doubtfully
So, I need only a count. Count all error lines in my log.
I have tried count_over_time, sum, rate, etc… Nothing helps.
One thing I ran into is that when the interval was very large the counts were not accurate, it was like it was dropping dome of the entries. This is why I set the max data points option, once I did that it was better unless I went to a really large interval.
Hi!
Thanks for sharing your solution, it helped me figure out how to solve a similar objective.
I was also reading the docs but while i couldn’t put the puzzle pieces together there, i remembered something that might be the cause of your issue with huge counts.
In your query you have a hardcoded interval of [1s] so if the inspected time range becomes larger, you are totalling up an increasing amount of 1s data points, growing towards and exceeding the max datapoints (thats why pushing that limit helped).
But that is where the $__interval variable can help. If you increase the time range to a month and would show that curve on a graph, you would have way more 1s data points than pixels on the screen, so grafana comes up with an interval of time that is roughly one datapoint per pixel based on the selected time range, and provides you that number in the $__interval to use.
Long story short, if you replace [1s] with $__interval your query will always total up a reasonably low amount of intermediate counts, and should not run in any datapoint limits (and maybe perform a little better too).
