I hope this query finds you well.
I am using Promtail to send a service’s logs to Loki.
In attempting to debug the service, I temporarily switched the service’s log level from
debug. Orders of magnitude more log entries were thus created. Then, I reverted to
An hour has passed and yet Promtail remains completely jammed.
My Promtail logs are full of entries such as:
caller=client.go:294 component=client host=22.214.171.124:5000 msg="error sending batch, will retry" status=429 error="server returned HTTP status 429 Too Many Requests (429): Ingestion rate limit exceeded (limit: 4194304 bytes/sec) while attempting to ingest '4328' lines totaling '1048569' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased"
caller=client.go:294 component=client host=126.96.36.199:5000 msg="error sending batch, will retry" status=-1 error="Post \"http://188.8.131.52:5000/loki/api/v1/push\": context deadline exceeded"
How can I discard these
debug entries and thus permit the steady flow of entries from Promtail to Loki? (I don’t care at all about losing any of the other yet-unsent log entries which are adjacent.)
PS: I searched elsewhere at length and could not find a treatment.