Loki - config for faster query performance

Grafana Loki
1

Hi,

I’m using loki 17.5.1 helm chart

I have 3 ingester replicas with 10gb and 2 cpu limit each. 6 querier replicas with 6gb and 2 cpu limit each. Max_concurrent is 16, Max_query_parallelism is 48, split query interval is 1h. I’m getting 4-5 seconds performance for 24 hrs query with this, but ingester and querier are hitting OOM or readiness probe issues and degrading.

Please advise the suitable config to achieve a stable, faster performance

2

max_concurrent should correspond to the number of CPU each querier has. So with 2 CPU on your queries I’d set max_concurrent to 2 or 4.

Curious, how much logs do you have roughly in a 24-hour window?

3

Loki is showing 15mil for 1 case and it takes approximately 15 seconds to load, with the config that i mentioned. Overall, we have 1-1.5TB log data per day.

Could you please share a config spec to get a faster query performance? Will reducing max_concurrent to 2 or 4 improve the query performance for my case?

4

It’s generally better to have smaller but more queriers than the other way around. I would recommend you to try:

  1. Changing max_concurrent to 2.
  2. Change your querier sizing from 6GB memory/2CPU to 4GBmemory/2CPU. Run more queriers, let’s say start with 12.
  3. Try it and see how it goes. Closely monitoring the querier resource utilization, and lower it down if desired so you can run more queriers.
  4. Take advantage of automatic scaling on kubernetes and scale your queries up when needed to optimize cost.
5

I have set max_concurrent to 4 and number of queriers to 12. Still, the 24hrs query is taking around 15 seconds for the first time.

6
  1. Do you know how much chunk data are actually pulled for the query?
  2. How much resources are your queriers using?
  3. Are you collecting metrics on your Loki cluster? Some of the metrics might be useful in terms of finding the bottleneck.
7
  1. 18Mil data is coming out in around 10 seconds
  2. split_query_intrval is set to 1h, max_query_parallelism is 48.
  3. We will enable the metrics dashboard .