Promptail HTTP status 429 Too Many Requests

[ragamx]

Hi

We have a Loki stack deployed in Kubernetes 3 Distributors, 3 Ingesters and 4 Promtail. Promtail so is throwing the following error

msg="error sending batch, will retry" status=429 error="server returned HTTP status 429 Too Many Requests (429): Maximum active stream limit exceeded, reduce the number of active streams
msg="Discovery receiver's channel was full so will retry the next cycle”

And the Ingesters

Maximum active stream limit exceeded, reduce the number of active streams (reduce labels or reduce label values)

I have set grpc_server_max_concurrent_streams to 0 (to be unlimited) and max_streams_per_user: 50 (default is 100) in Loki configuration but still getting those error messages

server:
  log_level: info
  # Must be set to 3100
  http_listen_port: 3100
  grpc_server_max_concurrent_streams: 0

limits_config:
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20
  max_concurrent_tail_requests: 20
  max_cache_freshness_per_query: 10m
  max_streams_per_user: 50

Any suggestion?

Thanks

[dror1212]

From my experience the error comes from

ingestion_rate_mb: 10
ingestion_burst_size_mb: 20

try making it higher (if your containers and network can handle it)

[ragamx]

I have increased these numbers for the last couple of days up to four times their initial value, but the error messages are still showing up. Thanks for the suggestion though.