Promtail's error logs with status 413

Grafana Loki
1

Recently I migrated loki from monolithic mode to distributed mode in our k8s cluster. After the migration, and reconfiguration of promtail (to new loki endpoint), I’m seeing the error messages from promtail frequently

level=error ts=2021-09-27T14:27:44.070198546Z caller=client.go:334 component=client host=loki-loki-distributed-gateway.loki.svc.cluster.local msg="final error sending batch" status=413 error="server returned HTTP status 413 Request Entity Too Large (413): <html>"

Note The configurations of loki in monolithic (single service) mode and distributed setup are same. which is as follows

server:
      http_listen_port: 3100
      grpc_server_max_recv_msg_size: 16777216
      grpc_server_max_send_msg_size: 16777216

ingester:
      lifecycler:
        ring:
          kvstore:
            store: memberlist
          replication_factor: 2
      chunk_retain_period: 1m
      chunk_idle_period: 6m
      max_transfer_retries: 0
      chunk_target_size: 1536000

limits_config:
      enforce_metric_name: false
      reject_old_samples: true
      reject_old_samples_max_age: 168h
      max_cache_freshness_per_query: 10m
      ingestion_rate_mb: 16
      ingestion_burst_size_mb: 18
      max_streams_per_user: 100000

Does grpc_server_max_recv_msg_size have to be increased? If so, it is working previously in single service setup.

Edit: We are also using long log lines in our apps and is this thing might be affecting it?

2

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.