Loki became very slow after upgrading from 2.3 to 2.73 (and only works in certain circumstances)

Grafana Loki
1

Hello!

About 2 month ago, our loki was upgraded from 2.3 to 2.7.3. Since then, our loki became very slow (before upgrade, it took about 0.5 seconds but now 10 seconds) and only runs under these 2 specific circumstances in our kubernetes environment:

it only works only if:

  1. The number of log entries in Loki >= query limit parameter, regardless of the query interval (e.g., “Start” time can be set to the last 30 days).

  2. Start time needs to be set to the last 24~26 hours at most, regardless of how many log entries exist in Loki. Even if the number of log entries in loki < limit, it still works.

None of the other cases work other than the 2 situations above (fails with timeout).

I tried increasing resources for loki, but that did not help.

this is the curl command I’m using:
curl -G -s “http://xxx:3100/loki/api/v1/query_range” --data-urlencode ‘query={job=“xxx”}’ --data-urlencode “start=${START_TIME}” --data-urlencode “limit=1000”

Has anyone experienced this issue before or does anyone know what could be the root cause?

Thank you very much.

2

I have never run Loki in monolithic mode or using a file system as storage, so I can’t really offer much practical advice. But I’d say try to set split_queries_by_interval to a bigger number (maybe 24h). If you are running monolithic mode you might actually get worse performance the more you split queries.

1 Like
3

This actually resolved the issue. Thank you very much, Tony

1 Like