I have set up some alerts in Grafana which is linked to a webhook. This integration works in terms of it alerting but when it is under the threshold to alert, it alerts for sometime (first notification), goes back to OK (sending a second notification in slack) and then alerts again (sending a third notification).
This seems to be a bug. There is nothing in the logs around the time to make it go back to ‘OK’.
I would like to know when the alert goes to OK so I do not want to remove that.
Used case example:
Alert threshold is set for a number ‘remaining’ is lower than 500.
It goes down from 520 to 480. It starts alerting and a message comes up in slack.
Multiple times a day, the alert goes to OK whilst still showing the current ‘remaining’ number (which is still below 500). Around the next evaluation time, it goes back to alerting, as it was in the first place.
What could be causing this and how can it be solved?
It is causing alerts to be spammed in a channel.