Grafana cloud otlp ingestion rate limits

Grafana Cloud
1

how can otel collector batch send rate be updated so that within grafana rate limits

otel-col.conf
processors:
batch:
send_batch_size: 8192 # Number of objects which will be sent in each batch
timeout: 60s # Time after which a batch will be sent anyway, regardless of size
send_batch_max_size: 0 # Upper limit of the batch size, 0=no upper limit

docker logs f7f6 --tail=10
2024-04-19T16:03:19.492Z info exporterhelper/retry_sender.go:129 Exporting failed. Will retry the request after interval. {“kind”: “exporter”, “data_type”: “metrics”, “name”: “otlphttp”, “error”: “Throttle (5s), error: error exporting items, request to https://otlp-gateway-prod-us-east-0.grafana.net/otlp/v1/metrics responded with HTTP Status Code 429”, “interval”: “20.100040796s”}
2024-04-19T16:03:24.432Z info exporterhelper/retry_sender.go:129 Exporting failed. Will retry the request after interval. {“kind”: “exporter”, “data_type”: “metrics”, “name”: “otlphttp”, “error”: “Throttle (4s), error: error exporting items, request to https://otlp-gateway-prod-us-east-0.grafana.net/otlp/v1/metrics responded with HTTP Status Code 429”, “interval”: “15.736277595s”}
2024-04-19T16:03:24.859Z info exporterhelper/retry_sender.go:129 Exporting failed. Will retry the request after interval. {“kind”: “exporter”, “data_type”: “traces”,“name”: “otlphttp”, “error”: “Throttle (0s), error: error exporting items, request to https://otlp-gateway-prod-us-east-0.grafana.net/otlp/v1/traces responded with HTTP Status Code 429, Message=RATE_LIMITED: ingestion rate limit (12500 bytes) exceeded while adding 8429284 bytes for user 863591, Details=”, “interval”: “4.611415953s”}
2024-04-19T16:03:26.194Z info exporterhelper/retry_sender.go:129 Exporting failed. Will retry the request after interval. {“kind”: “exporter”, “data_type”: “traces”,“name”: “otlphttp”, “error”: “Throttle (0s), error: error exporting items, request to https://otlp-gateway-prod-us-east-0.grafana.net/otlp/v1/traces responded with HTTP Status Code 429, Message=RATE_LIMITED: ingestion rate limit (12500 bytes) exceeded while adding 8388537 bytes for user 863591, Details=”, “interval”: “17.560590331s”}
2024-04-19T16:03:29.662Z info exporterhelper/retry_sender.go:129 Exporting failed. Will retry the request after interval. {“kind”: “exporter”, “data_type”: “traces”,“name”: “otlphttp”, “error”: “Throttle (0s), error: error exporting items, request to https://otlp-gateway-prod-us-east-0.grafana.net/otlp/v1/traces responded with HTTP Status Code 429, Message=RATE_LIMITED: ingestion rate limit (12500 bytes) exceeded while adding 8429284 bytes for user 863591, Details=”, “interval”: “4.174579532s”}
2024-04-19T16:03:32.586Z info exporterhelper/retry_sender.go:129 Exporting failed. Will retry the request after interval. {“kind”: “exporter”, “data_type”: “traces”,“name”: “otlphttp”, “error”: “Throttle (0s), error: error exporting items, request to https://otlp-gateway-prod-us-east-0.grafana.net/otlp/v1/traces responded with HTTP Status Code 429, Message=RATE_LIMITED: ingestion rate limit (12500 bytes) exceeded while adding 16566125 bytes for user 863591, Details=”, “interval”: “26.155451146s”}
2024-04-19T16:03:34.046Z info exporterhelper/retry_sender.go:129 Exporting failed. Will retry the request after interval. {“kind”: “exporter”, “data_type”: “traces”,“name”: “otlphttp”, “error”: “Throttle (0s), error: error exporting items, request to https://otlp-gateway-prod-us-east-0.grafana.net/otlp/v1/traces responded with HTTP Status Code 429, Message=RATE_LIMITED: ingestion rate limit (12500 bytes) exceeded while adding 8429284 bytes for user 863591, Details=”, “interval”: “5.940040753s”}
2024-04-19T16:03:40.011Z info exporterhelper/retry_sender.go:129 Exporting failed. Will retry the request after interval. {“kind”: “exporter”, “data_type”: “metrics”, “name”: “otlphttp”, “error”: “Throttle (4s), error: error exporting items, request to https://otlp-gateway-prod-us-east-0.grafana.net/otlp/v1/metrics responded with HTTP Status Code 429”, “interval”: “41.218907044s”}
2024-04-19T16:03:40.562Z info exporterhelper/retry_sender.go:129 Exporting failed. Will retry the request after interval. {“kind”: “exporter”, “data_type”: “metrics”, “name”: “otlphttp”, “error”: “Throttle (5s), error: error exporting items, request to https://otlp-gateway-prod-us-east-0.grafana.net/otlp/v1/metrics responded with HTTP Status Code 429”, “interval”: “19.178651282s”}
2024-04-19T16:03:41.199Z info exporterhelper/retry_sender.go:129 Exporting failed. Will retry the request after interval. {“kind”: “exporter”, “data_type”: “traces”,“name”: “otlphttp”, “error”: “Throttle (0s), error: error exporting items, request to https://otlp-gateway-prod-us-east-0.grafana.net/otlp/v1/traces responded with HTTP Status Code 429, Message=RATE_LIMITED: ingestion rate limit (12500 bytes) exceeded while adding 8429284 bytes for user 863591, Details=”, “interval”: “24.324467146s”}

2

12500 is more than batch size 8192

3

send_batch_size refers to the number of spans, log entries, and metric data points. It doesn’t refer to bytes. I believe there is currently no way to configure batching limits per bytes, but there might be soon.

For now, you could try setting a low enough batch limit to avoid this error most times. You can also check the batch processor’s metrics on localhost:8888/metrics. If the batch_send_size metric says that there aren’t that many units in the batch, but you’re still getting errors, then you may need to set the value even lower.

4

image
image1505×588 50.5 KB

image
image1110×825 124 KB

collector metrics dashboard not showing data ,sometimes can see some ,very inconsistent view of same data on these panels

after new deploy evening ,metrics ok but stopped during day time ,
could be due to excessive bytes during day workload ?

"error": "Throttle (0s), error: error exporting items, request to https://otlp-gateway-prod-us-east-0.grafana.net/otlp/v1/traces responded with HTTP Status Code 429, Message=RATE_LIMITED: ingestion rate limit (local: 12500 bytes, global: 525000 bytes) exceeded while adding 1847763 bytes for user 735249, Details=[]",
"interval": "3.972732488s"
5

updating our configs send_batch_size and send_batch_max_size to 512
how to estimate average batch size and timeout settings

ingestion rate limit (local: 12500 bytes, global: 525000 bytes) exceeded while adding 1847763 bytes