We are using grafana alerts with million of alert instances. From our checks, when all the alerts becoming firing (in a short period - we ran a stress test), the alertmanager isn’t sending all the alerts to the contact points (via webhook).
It seems that it sometimes misses alerts. So, my question, should I use polling in this case, and is it a good practice?
Why not many technologies actually use the polling api from grafana? It seems to be more robust and “reliable”, but a the same time, more resource heavy on grafana, which is somewhat ok.
Welcome @zacu016 to the community
I reproduced this issue with 1000 alert instances. A key root cause is
SQLite database lock contention.
Grafana logs under 1000 alert instances:
database is locked (SQLITE_BUSY)
Tick dropped because alert rule evaluation is too slow
SQLite cannot handle concurrent writes from thousands of alert state changes
Grafana drops entire evaluation cycles when DB is locked
Alerts are missing because they were never evaluated, not because webhook missed them
I tested this: pushed 1000 alert instances simultaneously → only 909 received by webhook → logs confirmed SQLite lock as cause
Regarding polling: won’t help → same overloaded backend, same DB bottleneck.
Try Switch to PostgreSQL or MySQL. Grafana’s own docs recommend this for any production deployment. SQLite is for test/small setups only.
More like I reproduced it with postgress with 500k alert instances with evaluation interval of 2 minutes.
We are already use postgress with cloudnativepg and connection pooling.
Thanks for the context → our test was on default SQLite with 1000 instances, so it doesn’t apply to your setup.
Since you’re already on PostgreSQL with CloudNativePG and connection pooling, can you check your Grafana logs for:
Tick dropped because alert rule evaluation is too slow
If this appears, Grafana is skipping evaluation cycles at 500k scale. That would be the confirmed root cause for your setup.
please show your alerting metrics over time:
and USE metrics for your Grafana server instance and db instance
