Discarding Promtail log entries en masse?

Grafana Loki
1

Hello:

I hope this query finds you well.

I am using Promtail to send a service’s logs to Loki.

In attempting to debug the service, I temporarily switched the service’s log level from info to debug. Orders of magnitude more log entries were thus created. Then, I reverted to info.

An hour has passed and yet Promtail remains completely jammed.

My Promtail logs are full of entries such as:

caller=client.go:294 component=client host=172.3.4.5:5000 msg="error sending batch, will retry" status=429 error="server returned HTTP status 429 Too Many Requests (429): Ingestion rate limit exceeded (limit: 4194304 bytes/sec) while attempting to ingest '4328' lines totaling '1048569' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased"

and

caller=client.go:294 component=client host=172.3.4.5:5000 msg="error sending batch, will retry" status=-1 error="Post \"http://172.3.4.5:5000/loki/api/v1/push\": context deadline exceeded"

How can I discard these debug entries and thus permit the steady flow of entries from Promtail to Loki? (I don’t care at all about losing any of the other yet-unsent log entries which are adjacent.)

|
Graffana

PS: I searched elsewhere at length and could not find a treatment.

1 Like
2

You have a couple options, but since you sent this many days ago this will probably find you a little too late.

Your loki server is rate limiting you, rate limited sends from promtail are retried, so as long as your incoming log stream isn’t too large eventually it should catch up.

Or you can increase the rate limit in Loki, check out the limits config for ingestion_rate_mb and ingestion_burst_size_mb

These limits mostly exist to protect your Loki server from being tipped over by a client, but even a single instance Loki on most machines can likely handle having that limit increased.

2 Likes
3

Hi @ewelch,
regarding ingestion_rate_mb, I am currently deploying loki in k8s using loki-distributed helm chart, but I don’t see a difference using global or local ingestion_rate_strategy. I still get Too Many Requests (429) error regardless.

For example I am having 6 distributors, with ingestion_rate_mb: 60, with global mode, if my understanding is correct, the limit for each distributor should be 10mb/sec. That would be equivalent to ingestion_rate_mb: 10 for local mode.
Although, promtail seems to be sending logs at around a 10mb/sec rate

    limits_config:
      ingestion_rate_strategy: global
      ingestion_rate_mb: 60
      ingestion_burst_size_mb: 85
----
Ingestion rate limit exceeded for user fake (limit: 10485760 bytes/sec)

promtail
promtail519×1007 25.3 KB

Could you please share your knowledge why this happens?

Thank you in advance.

4

Which component is reporting the rate limit error?

5

This comes from promtail