Hello, there.
I’m currently using Grafana Cloud and I’m trying to create a heatmap to display the duration of HTTP requests.
I am ingesting logs from Caddy, and they look like this:
{"level":"error","ts":1725022941.7703881,"logger":"http.log.access.log1","msg":"handled request","request":{"remote_ip":"192.168.33.99","remote_port":"53583","client_ip":"192.168.33.99","proto":"HTTP/1.1","method":"GET","host":"test.hoth.internal","uri":"/","headers":{"User-Agent":["curl/8.7.1"],"Accept":["*/*"]}},"bytes_read":0,"user_id":"","duration":0.001064365,"size":0,"status":502,"resp_headers":{"Server":["Caddy"]}}
As you can see there is a filed called duration that track how long it took for the request to be satisfied.
I’m trying to extract all of this information and plot them into a heatmap.
I came up with the following query, which seems to correctly get the data
sum by (duration_bucket)
(
count_over_time({filename="/home/rhaidiz/caddy/logs/test.hoth.internal.log"}
| json
| label_format duration_bucket=`{{ if hasPrefix "0." .duration}}0{{ else if hasPrefix "1." .duration }}1{{ else }}inf{{end}}`
[$__auto]
))
Essentially what I’m trying to do is manually defining duration bucket and check the prefix of the duration to see in which bucket they fit. This seems to work and in the explorer panel I see that the logs are grouped approprietly.
If I plot them into a heatmap I can also see that something is showing (see screenshot below).
However, one thing that I cannot understand is how to have better time internal. I though about expanding the query step to 10m, but if I do it I don’t see anything plotted in the heatmap anymore.
I feel like I’m missing something very basic here, any idea what that might be?