-
What Grafana version and what operating system are you using? Version 12.1.0
-
What are you trying to achieve? Loading page Grafana alerting.
-
How are you trying to achieve it?
-
What happened? Alerts in Grafana have no states.
-
What did you expect to happen? When my data source responds slowly, alerts report an error, and alerts from other sources are not affected.
I’m having a problem with alerts in Grafana.
My installation is a standalone OSS version 12.1.0.
I’m using MySQL as the backend.
I retrieve most of my data from an external data source, which is ElsticSearch.
Sometimes, when Elastic responds slowly, Grafana has trouble loading alert states. Instead of setting an error to the alert state, the alert page doesn’t load, creating a spinning wheel effect.
During this time, it seems that other alerts from other sources, such as MySQL, Prometheus, and Zabbix, are also unable to retrieve their state and are resetting. This state persists until Elastic responds in a timely manner and I restart Grafana.
Is there any way to fix this problem? In my opinion, if an alert hasn’t received a response within the expected time, it should assume an Error state and maintain it. And in this case it affects other alerts not related to Elastic.
Two things worth checking:
Set an explicit HTTP timeout on your Elasticsearch datasource (Connections → your Elasticsearch datasource → HTTP settings → Timeout). Try 10–15s so slow requests fail faster.
Also check the evaluation_timeout setting under [unified_alerting] in grafana.ini. If it is set unusually high, try lowering it and see whether that changes the behavior.
Since alerts using MySQL, Prometheus, and Zabbix also stop updating until Grafana is restarted, if reducing the timeouts doesn’t change the behavior, then enable debug logging and capture the relevant alert logs while the issue occurs will help determine whether this is a configuration issue or a Grafana bug.
Thanks for your replay.
The first thing I did earlier was to change all Elastic data source to reduce the timeout to 10 seconds.
This didn’t yield a satisfactory result. Therefore, I also changed the settings in the grafana.ini file for [unified_alerting]. My currents settings:
evaluation_timeout = 10s
max_attempts=1
min_interval=30s
scheduler_tick_interval=10s
Are my setting correct?
I don’t see anything alarming in the logs other than an error related to timeout Elastic.
I also see warning log like “Tick dropped because alert rule is too slow”. And suddenly, alert_rule gets a message that’s sending an alert completion message from a completely different source, e.g., Prometheus. It’s as if the alerts have restarted.
I added, at this time grafana API monitoring but address localhost/api/prometheus/grafana/api/v1/rules is nog responding, it gets a timeout.
l don’t know what to look for or where the problem might be.
Your timeout settings look reasonable, and since reducing both the Elasticsearch datasource timeout and evaluation_timeout didn’t change the behavior, it doesn’t appear that adjusting those settings alone resolves the problem.
The key symptom is that /api/prometheus/grafana/api/v1/rules also times out and alert rules from unrelated datasources stop updating. A slow Elasticsearch query would normally be expected to affect only the alert rules that depend on it, rather than causing unrelated alert rules and the alerting API to become unresponsive.
Could you temporarily pause all Elasticsearch-backed alert rules and check whether the Prometheus, MySQL, and Zabbix alert rules continue evaluating normally? That will help determine whether the Elasticsearch alert evaluations are triggering the problem or whether this is a broader Grafana issue.
