High CPU usage by Promtail

Grafana Loki
1

Hi,

I’ve noticed that Promtail can use a lot of CPU.

In one particular case, I’ve observed almost 800% of CPU when monitoring 85k log paths (number of rows in positions.yaml file). If positions.yaml contains over 600k log paths, I’ve noticed almost 2000% of CPU.

We use file-based service discovery with:

  • refresh_interval: 5s
  • positions sync_period: 5s
  • client batchwait: 5s

I know that this is a lot of logs, but still, is this expected?

If yes, can anyone have some ideas how to reduce it?

I’ve noticed that after increasing sync_period and refresh_interval to 30s, CPU usage went down, but still was high. We’ve also noticed, that the number of rows in positions.yaml file have a direct impact on CPU usage. We’ve added an automated procedure to remove unnecessary rows from positions.yaml (of logs that we know nothing will be written to). This also helped a bit.

Thanks in advance for any ideas how to further reduce Promtail CPU usage.

2

I think that this is hard question to answer without knowing how much raw compute you’re dealing with. Can you give us more specific numbers?

Here is a twitter post about 52GB/s Loki log filtering using ~140 CPUs at roughly 400MB/s per CPU:

3

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.