Error: "Error sending batch, will retry" - Ingestion rate limit exceeded in Loki

Grafana Loki
1

Hi Community,
We are encountering the following error when attempting to send logs to Loki:

msg=“error sending batch, will retry” component_path=/ component_id=loki.write.logs_alloy component=client host=loki:3100 status=429 tenant=“” error=“server returned HTTP status 429 Too Many Requests (429): Ingestion rate limit exceeded for user fake (limit: 4194304 bytes/sec) while attempting to ingest ‘4213’ lines totaling ‘1048378’ bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased”

We have 200+ pods, each representing a different microservice, and they all generate logs. These logs are mounted to a host path (on-prem server with a single-node cluster). The logs are stored under /var/log/CustomerLog/. The ingestion rate limit is being exceeded, resulting in this error message. We request advice on how to manage or reduce the ingestion rate, whether it’s possible to increase the ingestion limit to accommodate the log volume, and any best practices for scaling Loki in a single-node cluster setup to handle this load.

any suggestion @tonyswumac @yosiasz

2

How are these logs being pushed to Loki?

I is Loki a single server setup?

3

Thanks for response @yosiasz
yes, Loki is single server setup
we are using alloy as agent
and this config of alloy
logging {
level = “debug”
format = “logfmt”
}

loki.echo “alloy_log_scraper” {}

loki.write “logs_alloy” {
endpoint {
url = “http://loki:3100/loki/api/v1/push”
}
}

local.file_match “logs_alloy_log_scraper” {
path_targets = [{
address = “localhost”,
path = “/var/log/CustomerLog//.log”,
}]
}

loki.process “logs_alloy_log_scraper” {
forward_to = [loki.write.logs_alloy.receiver]

stage.regex {
expression = “/var/log/CustomerLog/(?P[^/]+)/(?P[^.]+).log”
source = “filename”
}

stage.regex {
expression = “DateTime: (?P[^|]+) \| FunctionsStructure: (?P[^|]+) \| Signals: (?P[^|]+) \| Value: (?P[^|]+) \| Quality: (?P[^|]+) \| PlaceHolders: (?P.*)”
}

stage.timestamp {
source = “DateTime”
format = “RFC3339Nano”
location = “UTC”
}

stage.json {
expressions = {
DateTime = “DateTime”,
FunctionsStructure = “FunctionsStructure”,
Signal = “Signal”,
Value = “Value”,
Quality = “Quality”,
PlaceHolders = “PlaceHolders”,
}
source = “expression”
}
}

loki.source.file “logs_alloy_log_scraper” {
targets = local.file_match.logs_alloy_log_scraper.targets
forward_to = [loki.process.logs_alloy_log_scraper.receiver]
}

4

look at this from grafana slack Ask AI channel

Also read this

5

in loki ```
deploymentMode: DistributedordeploymentMode: SimpleScalable`
will those support filesystem since we have single node cluster
and we not using cloud storges, like s3 etc.
we are using host level mounts

6

as listed in my previous post you should try the 1st approach first

loki.write "my_loki" {
     endpoint {
       url = "http://loki:3100/loki/api/v1/push"
     }
     sending_queue {
       enabled = true
       num_consumers = 10
       queue_size = 5000
     }
   }

or

loki.write "my_loki" {
     ...
     batch {
       timeout = "1s"
       send_batch_size = 1000
     }
   }

read the docs on batch and sending_queue if those dont do it then do the deployment models