Loki full CPU for a long time and need restart to work again

[christoforosd]

Hi, random times i get full CPU for a long time and every stop working, i need to restart loki to work. IT was full for 15 min before restart container.

I have one container only, with boltdb and S3.

Any idea what i need to search to find the issue? Or maybe why is happening?

[tonyswumac]

What is your log volume looking like? Perhaps it’s time to look into simple scalable deployment.

But if you want to just run a single instance, I’d recommend you to try and reduce the number of chunks written to S3, and avoid splitting query too much.

[christoforosd]

how can i find my volume?

Is to small maybe a 5-6 GB per day. Now is one EC2 with 8 CPU, i am planning to have a small scalable but i cannot now but i need to find the issue with this but no luck

Average per day is ~25% CPU and some spikes to 50-60%.

[tonyswumac]

1. Check your storage and see how many files you have a day. If you are running a single instance I’d try to aim for having as few files as possible.

2. Reduce query split.

[christoforosd]

hi Tony, thank you for your reply

1. storage files:
   from S3 bucket metrics: 30000 (not sure how i can check from loki metrics)

If you are running a single instance I’d try to aim for having as few files as possible.

How can you have fewer files? delete the old?

2. Reduce query split.

I will try change this (not sure if is this )

limits_config:
split_queries_by_interval

[tonyswumac]

There are a couple of configurations that you can use to control how many chunks get written, such as max_chunk_age, chunk_idle_period, and chunk_target_size. Make sure to also adjust query_ingesters_within to match.

As for splitting queries, split_queries_by_interval is what you want. You can change that to 12h or 24h for example and see how it performs.