Rpc error: code = Code(429) - ingester

Grafana Loki
1

level=warn ts=2023-05-24T16:01:03.767298774Z caller=grpc_logging.go:43 method=/logproto.Pusher/Push duration=965.209µs err=“rpc error: code = Code(429) desc = Maximum active stream limit exceeded, reduce the number of active streams (reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be increased” msg=gRPC

I am getting the above exception in loki ingester.
Please help to fix these issue.

2

How many labels do you have in your logs?

3

Total 8 labels i have configured in promtail pipelines stage.

4

limits_config:
enforce_metric_name: false
ingestion_rate_strategy: local
ingestion_burst_size_mb: 128
ingestion_rate_mb: 128
per_stream_rate_limit: 128MB
per_stream_rate_limit_burst: 128MB
reject_old_samples: true
reject_old_samples_max_age: 15m
# default = 1m
max_cache_freshness_per_query: 5m
max_concurrent_tail_requests: 100
max_global_streams_per_user: 100000
max_streams_per_user: 100000
# default = 30m (decrease to increase parallel processing)
split_queries_by_interval: 24h
retention_period: 2160h

grpc_server_max_recv_msg_size: 4294967296
grpc_server_max_send_msg_size: 4294967296

5

What are those labels? How many potential values for each label?

Also see Best practices | Grafana Loki documentation.

6

i am extracting the keys from json log stream and adding into labels. All those 8 labels i am using to build the dashboards.
ie - http_status_code, response, total_time, service_name

7

Then you are probably overdoing labels. Please see the link in my previous reply.

total_time is probably not a good label, neither is status code and response.

8

let suppose in grafana status code wise i need to take count of total request in grafana.
in my case i am using below query.
sum by(http_status_code) (count_over_time({job=“applogs”, service_name=“http-server”}[$__interval]))

can u help me to make the same output if i dont put any labels ie http_status_code and service_name ?

9

You can use logql to parse log lines. see LogQL: Log query language | Grafana Loki documentation.

10

can u convert my query and share me. it will be helpful for me

11

Do you have an example of your log line?

12

{
“level”: “info”,
“job”: “applogs”,
“service_name”: “http-server”,
“http_status_code”: 200,
“msg”: “success”,
“scip”: “x.x.x.x”,
“ts”: “xxxxxxxxx”
}

13 
{<LABEL_SELECTOR>} | JSON

should be all you need.