429 Too Many Requests Maximum active stream limit exceeded, reduce the number of active streams

Grafana Loki
1

Hi…

i’m trying to configure loki cluster using loki. (simple scalable mode)
but when i try to many logs to loki, 429 Too Many Requests occurs.

i try to find solution in community and modified limits_config.
but it still not working to mee… any idea you haves.?

global:
  dnsService: core-dns

server:
  #Increasing the value of grpc_server_max_recv_msg_size allows larger messages to be received by the server, accommodating scenarios where large log payloads or other data need to be transmitted.
  grpc_server_max_recv_msg_size: 8388608 # default 4194304
  #This parameter sets the maximum size (in bytes) for outgoing gRPC messages that Loki's server can send.
  grpc_server_max_send_msg_size: 8388608 # default 4194304
  http_server_read_timeout: 300s # default 30, Reade timeout for HTTP server
  http_server_write_timeout: 300s # default 30, Write timeout for HTTP server
loki:
  schemaConfig:
    configs:
      - from: 2024-06-13
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h
  tracing:
    enabled: true
  persistence:
    enabled: true
    storageClassName: ssd
    size: 10Gi

  storage:
    type: s3
    bucketNames:
      chunks: "loki"
      ruler: "loki"
      admin: "loki"
    s3:
      endpoint: endpoint
      accessKeyId: accesskey
      secretAccessKey: secretKey
      signatureVersion: v4
      s3ForcePathStyle: true
      insecure: true
      http_config:
        insecure_skip_verify: true

deploymentMode: SimpleScalable

backend:
  replicas: 3
  resources:
    limits:
      cpu: "1"
      memory: "1Gi"
read:
  replicas: 3
  resources:
    limits:
      cpu: "1"
      memory: "1Gi"
write:
  replicas: 6
  resources:
    limits:
      cpu: "4"
      memory: "4Gi"

lokiCanary:
  resources:
    limits:
      cpu: "0.5"
      memory: "500Mi"

chunksCache:
  allocatedMemory: 1024

# https://grafana.com/docs/loki/latest/configure/#limits_config
# modify if limit error occur (429 Too Many Requests)
limits_config:
  retention_period: 240h
  max_entries_limit_per_query: 5000
  max_chunks_per_query: 200000
  reject_old_samples: true
  reject_old_samples_max_age: 24h
  ingestion_rate_mb: 1000
  ingestion_burst_size_mb: 1500
  max_query_series: 100000
  max_label_value_length: 10000
  max_label_names_per_series: 1000
  max_line_size: 1024KB
  max_line_size_truncate: true
  max_streams_per_user: 0
  max_global_streams_per_user: 0
  per_stream_rate_limit: 512MB
  per_stream_rate_limit_burst: 1024MB

# Disable minio storage
minio:
  enabled: false

# Zero out replica counts of other deployment modes
singleBinary:
  replicas: 0

ingester:
  replicas: 0
  ingester:
    chunk_encoding: gzip
querier:
  replicas: 0
  querier:
    max_concurrent: 4
queryFrontend:
  replicas: 0
queryScheduler:
  replicas: 0
distributor:
  replicas: 0
indexGateway:
  replicas: 0
bloomCompactor:
  replicas: 0
bloomGateway:
  replicas: 0
2

message is following…

Too Many Requests Maximum active stream limit exceeded, reduce the number of active streams (reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be increased, user:

3

do not use trace_id as label… high cadinality