My service is logging stats like request latency to Elasticsearch that I want to display in Grafana. I’m trying to make singlestat panels that reflect the average and 95-percentile latency for all requests over the displayed time period, but I can’t figure out a way to do it accurately.
The problem is that if I group by Date Histogram and then aggregate again by picking “avg” from the Options tab, I end up getting an average of averages of latency in each time bucket, which usually isn’t the same as the average latency over the entire time period.
I tried to get around this by defining a custom interval that spanned the entire time range shown, but Grafana doesn’t allow an interval step count of 1.
I also tried to group everything into the same bucket by grouping by a Term that I knew would be the same across all documents, but I get the following error:
TypeError: Cannot read property 'toString' of undefined
at a.isMsResolutionNeeded (https://grafana.tubularlabs.net/public/app/boot.6d63bd8b.js:21:3590)
at new a (https://grafana.tubularlabs.net/public/app/boot.6d63bd8b.js:21:934)
at b.seriesHandler (https://grafana.tubularlabs.net/public/app/boot.6d63bd8b.js:20:23648)
at Array.map (native)
at b.onDataReceived (https://grafana.tubularlabs.net/public/app/boot.6d63bd8b.js:20:23515)
at f.emit (https://grafana.tubularlabs.net/public/app/boot.6d63bd8b.js:52:2928)
at a.emit (https://grafana.tubularlabs.net/public/app/boot.6d63bd8b.js:52:5256)
at b.handleQueryResult (https://grafana.tubularlabs.net/public/app/boot.6d63bd8b.js:46:4942)
at g (https://grafana.tubularlabs.net/public/app/boot.6d63bd8b.js:42:13902)
at https://grafana.tubularlabs.net/public/app/boot.6d63bd8b.js:42:14074
Is there something I’m missing? Is there a way tell Grafana that I don’t want to group-by anything at all for singelstat metrics?