I have some alerts which most of the time don’t have data, so I’ve put a “NoData” setting to “OK” in the configuration for them. And everything worked well and I didn’t receive NoData messages from these alerts until I updated Grafana from 9.0.3 to 9.3.6.
After the update I’ve started to receive other type of NoData messages, which are not related to the setting “NoData=OK”, because I didn’t change that.
So my question is:
Why “NoData = OK” setting is not working with “grafana_state_reason: NoData” alerts?
What changed after the update?
I am using Telegram as a contact point and ClickHouse as a datasource.
Grafana v9.5.2 (cfcea75916) the problem still persists.
Unfortunatly, swtiching Grafana to the debug mode gave no useful information. When I am trying to evaluate query on the Edit page of the alert everything is okay.
It seemed like this issue was relevant.
And I wrote a comment there, but got no answer…
Only a link to the documentation, where no explanation of how the grafana_state_reason: NoData state works.
This state reason is sent to distinguish between situations when the transition Alerting -> Normal happens due to the metric going below the threshold or due to missing data points.
This is what happens in this case.
Now, to stop receiving such messages users have two options:
disable resolve notifications. Each contact point integration has this setting. It can be changed on “Contact points” page. This way, only notifications about firing Alerts will be sent.
use custom notification template and drop the annotation grafana_state_reason
