How to restrict alerting query concurrency? (Rate limiter)

Grafana Alerting
,
1

Hi, all
I have encountered the problem about alerting. Just how to restrict alerting query concurrency?
For example, we have 20000 alert query will be executed every one minute. But these 20000 query simultaneous to execute may result in corresponding datasource(ex. ClickHouse) high concurrency problem so that can not get response and result int datasource shut down.

So if we can control concurrency of alerting, like first execute 10000 queries. once completed to execute other 10000 queries, we think that can protect datasource concurrency too high.

Does have anyone understand how to process it? and does the Grafana support it? Thank you.

2

Try to configure this config properly:

[unified_alerting]
# Disables the smoothing of alert evaluations across their evaluation window.
# Rules will evaluate in sync.
disable_jitter = false

Theory: 20k alerts with 1 min interval = 333 req/sec if they are smoothed in the ideal way. It is not rate limiting, but smoothing (spreading across whole evaluation window)

3

@jangaraj So we need try to enable this configuration? The default value is false.