Show better Error Message when Loki hits a Timeout

Grafana Loki
1

I have a .NET application that uses Serilog and a Grafana-Loki sink to write the Logs into Loki.

Then in Grafana I have a dashboard where my colleagues and I can query those logs with various filters. It often happens that someone enters a time range that is too big for loki, and the query seems to timeout. In those cases, the tables simply says NO DATA.

It has happened more than enough times that someone thought there were NO LOGS AT ALL, instead of trying again and seeing that there are actually logs, if they had just minimised the time range.

Therefore, I’d want to implement some sort of error message pop-up or something like that, which makes the user clear that it was a TIMEOUT, and there might still be data available. It’s pretty important because a big part of our debugging process depends on searching logs.

Maybe there’s even a way to improve performance some way? I have added 8GB of RAM to the Loki Container, of which I have only ever seen it use around 3-4. I have also increased the Timeout to 120 Seconds.

Any input is appreciated, thanks in advance!

2

I can’t speak to the error message, but if you are looking to improve query performance you will have to consider micro service mode and implement query splitting. In your use case thought it might be a bit overkill.