Maximum active stream limit exceeded, reduce the number of active streams

[afreyermuth98]

Hello, I’m trying to collect logs of an apache server with promtail and sending it to a remote host with loki and grafana installed. The problem is that I have a lot of logs and it seems that Loki cannot handle this.
Indeed, here is the error that promtail gives me :

level=warn ts=2021-03-05T09:03:14.685601189Z caller=client.go:322 component=client host=MyMonitoringDomain:3100 msg="error sending batch, will retry" status=429 error="server returned HTTP status 429 Too Many Requests (429): Maximum active stream limit exceeded, reduce the number of active streams (reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be increased"

Here is my promtail conf :
server:
http_listen_port: 9080
grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
    - url: http://MyMonitoringDomain.com:3100/loki/api/v1/push

scrape_configs:
- job_name: apache
  pipeline_stages:
    - regex:
        expression: "^(?P<ip>\\S+) (?P<identd>\\S+) (?P<user>\\S+) \\[(?P<timestamp>[\\w:/]+\\s[+\\-]\\d{4})\\] \"(?P<action>\\S+)\\s?(?P<path>\\S+)?\\s?(?P<protocol>\\S+)?\" (?P<status_code>\\d{3}|-) (?P<size>\\d+|-)\\s?\"?(?P<referer>[^\"]*)\"?\\s?\"?(?P<useragent>[^\"]*)?\" (?P<microseconds>\\d+)?$"
    - labels:
       path:
       microseconds:

  static_configs:
  - targets:
      - localhost
    labels:
      job: apache
      __path__: /var/log/apache2/access.log

And my conf of loki is the base config (unchanged)
Would be really happy if someone can help me with this !

[ewelch]

@afreyermuth98

 - labels:
       path:
       microseconds:

This is why…

You almost never want to extract content from log lines and put it in labels and you definitely never want to extract something with high cardinality like path could potentially have or microseconds which is basically unbounded. This will result in creating a stream for every message you sent and will hit that limit almost immediately.

This blog post explains in some more detail how to use labels in a way that makes Loki happy.