Mute Timing caused email notifications to stop completely

After upgrading Grafana to 8.4.2 (on Debian 10) I switched from [alerting] to [unified_alerting] which converted my existing alert to the new Grafana 8 Alerting alert rule. At this point the converted alert rule was sending out emails just fine, just like before the switch.

Then in Notification Policies I created a new Mute Timing. (at this point my alert was still working, sending out [FIRING] and [RESOLVED] emails).

Then I edited the policy for my alert and added my new Mute Timing. From this point on sending out emails stopped, regardless of outside or inside of the Mute Timing interval.

In attempt to make the email notifications work again I removed the Mute Timing from the policy, but it did not help. I ended up with an alert rule that was supposed to send out emails, but did not.

Is this a bug, or have I overlooked something?

Thank you,
Regards,
Oskar

How to reproduce:

  • install Grafana 8.4.3 with [alerting] enabled and [unified_alerting] disabled
  • add a Prometheus Data source
  • add an Alert/Notification channel of Email type
  • create a dashboard, add a Graph(old) type of panel with this query: ((minute() >= bool 0) * (minute() < bool 10)) + ((minute() >= bool 20) * (minute() < bool 30)) + ((minute() >= bool 40) * (minute() < bool 50))
  • create an alert with
    • Condition: WHEN last() of query(A, 5m, now) IS ABOVE 0
    • Notifications: Send to Email

This will alert for 10 minutes, resolve for 10 minutes, alert for 10 minute, resolve for 10 minutes… sending a “FIRING” and “RESOLVED” email every time the state changes.

Then migrate from [alerting] to [unified_alerting], define a new Mute Timing, and add it to the converted alert. The state will keep flapping every 10 minutes, but there will be no “FIRING” and “RESOLVED” emails anymore.

Is this a bug?

Thanks,
Oskar

Is any grafana developer able to reproduce this bug?