Loki Log Rate Limit

Grafana Loki
1

Good morning,

I am seeing an issue when adding a limit configuration to my promtail configuration file. We have some rather large servers that we need to install promtail on but installing and enabling promtail would instantly overload the promtail server with our log ingestion. I researched into different methods of limiting the logs that are sent through but my approach doesn’t seem to work. Here is my configuration, and below that is the error I’m receiving:

I’m running the OSS version of Grafana. I just need a recommendation on what configuration block I can use in my promtail configuration to limit the rate of log ingestion sent out and into my Loki server. I use Grafana to visualize these logs sent to Loki.

When I run promtail on a large server without a limit config, I am facing downtime on my server. For Loki/Promtail v2.9.5, what is a good log limit configuration I can use in my promtail configuration?

Something like this:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://test.com:3100/loki/api/v1/push

scrape_configs:
- job_name: provider
  static_configs:
  - targets:
      - localhost
    labels:
      job: providerlogs
      __path__: /opt/xxx/xxx/log/xxx.log
      server: xxx-app

- limit:
    rate: 10
    burst: 10

For me it returns this error and I’m not sure if it’s just not valid to use anymore, and if not what is a good substitute?

The error:

Unable to parse config: /etc/promtail/promtail-local-config.yaml: yaml: unmarshal errors:
promtail-linux-amd64: line 53: field limit not found in type scrapeconfig.plain

Thanks!

2

That’s because limits are not part of scrape_configs. And limits doesn’t exist at all (that looks like ChatGPT advice :-()

Always check doc first:

So example:

...
scrape_configs:
...
limits_config:   
  readline_rate_enabled: true
  readline_rate: 2500
  readline_burst: 5000
  ...

Please use formatting for your code, config, logs snippets.

3

Sure, I will give this a try. I will say that I have seen this documentation before and it is not chatgpt advice, I grabbed it from this link: limit | Grafana Loki documentation

Thank you for the advice though, let me give this a go.

4

Looks like it was able to work. It’s interesting I tried your approach a couple weeks ago before I was tasked with another item and I was getting an error.

I assume now that was probably because I had it formatted incorrectly. I wish I saved the error but oh well. It’s nice to see it working. Now I just need to figure out which values for my rate will be best.

Thanks!

5

OK, you mixed (Loki) doc with different product (Promtail). Loki != Promtail. :+1:

1 Like
6

:man_facepalming:

They don’t call me a senior engineer for nothing right! Lol. Thanks for the callout.

1 Like