Discarding Promtail log entries en masse?


I hope this query finds you well.

I am using Promtail to send a service’s logs to Loki.

In attempting to debug the service, I temporarily switched the service’s log level from info to debug. Orders of magnitude more log entries were thus created. Then, I reverted to info.

An hour has passed and yet Promtail remains completely jammed.

My Promtail logs are full of entries such as:

caller=client.go:294 component=client host= msg="error sending batch, will retry" status=429 error="server returned HTTP status 429 Too Many Requests (429): Ingestion rate limit exceeded (limit: 4194304 bytes/sec) while attempting to ingest '4328' lines totaling '1048569' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased"


caller=client.go:294 component=client host= msg="error sending batch, will retry" status=-1 error="Post \"\": context deadline exceeded"

How can I discard these debug entries and thus permit the steady flow of entries from Promtail to Loki? (I don’t care at all about losing any of the other yet-unsent log entries which are adjacent.)


PS: I searched elsewhere at length and could not find a treatment.

You have a couple options, but since you sent this many days ago this will probably find you a little too late.

Your loki server is rate limiting you, rate limited sends from promtail are retried, so as long as your incoming log stream isn’t too large eventually it should catch up.

Or you can increase the rate limit in Loki, check out the limits config for ingestion_rate_mb and ingestion_burst_size_mb

These limits mostly exist to protect your Loki server from being tipped over by a client, but even a single instance Loki on most machines can likely handle having that limit increased.