False alerts every 15 minutes, with a set interval of 1 hour

andr3wk · July 5, 2023, 2:58pm

The screenshots show that the first alert comes, then after 5 minutes a message about its resolution comes, after 10 minutes the cycle repeats and so on for an hour.
In the alert setting, an interval of 1 hour is set. Alert resolution cannot come after 5 minutes, nor can a re-alert after 15 minutes. As you can see in the screenshots, the threshold is exceeded 1 time, then there are no errors, but there are alerts.

This behavior is repeated many times on different alerts. The number of errors in all alerts is the same as in the first alert. The screenshots show the alert settings. The condition for triggering an alert can be seen in the screenshot.

False_alerts_settings_2

Alerts are set to email notification. Grafana version is v9.0.0. Promtail source - logs via Loki-Logs that are pulled from a common JSON file with logs. This behavior appears randomly, the rest of the time alerts do not give false positives. Please help me to deal with this problem.

georgerobinson · July 5, 2023, 3:14pm

Hi! This screenshot show an interval of 1 minute, not 1 hour? Is that intentional?

False_alerts_settings_2

andr3wk · July 6, 2023, 6:52am

Hi! This thing indicates that it checks every minute to see if there are any pending alerts. If in 5 minutes they have not disappeared, then the alert becomes firing. But the one hour is set here.

In normal mode, if an alert has been triggered, it lasts about an hour and during this time there can be no messages that the alert is resolved, and even more so, the message of a new alert.