Grafana 8 Alert not working as expected

[libarton]

Hi,
I have upgraded Grafana from 7.4 to 8.1.5 with the new alerting mechanism.
The ‘new alerting’ not working properly (also not so intuitive imo)
There is no testing button in order to test connection (should be coming out soon afaik)

• What Grafana version and what operating system are you using?
  Grafana 8.1.5 - docker image - alpine

• What are you trying to achieve?
  I’m trying to test a new alert (using the new ngalert)

• How are you trying to achieve it?
  Defined dashboard
  Defined contact point (slack channel → webhook is properly configured as ‘curl’ command running successfully)
  Defined Notification policy
  Defined proper labelling matching between the notification-policy and alert labelling.
  Triggered the alert → ALERT WAS FIRING PROPERLY
  But did not send the alert to slack

• What happened?
  Alert was not sent to slack notification

• What did you expect to happen?
  Expected to receive slack notification

• Can you copy/paste the configuration(s) that you are having problems with?
  new NGALERTS not working properly

• Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
  No errors

• Did you follow any online instructions? If so, what is the URL?
  Create Grafana managed alert rule | Grafana documentation

How can I make sure connection with slack is ok ?
What exactly should be with the labeling in notification policy? I see every alert get label “alertname” → should it be part of the notification policy ?

Thanks.

[nbfowler]

I think this could explain your issue: [ngalert] Missing alert notifications in Grafana 8.1.3 and 8.1.4 · Issue #39295 · grafana/grafana · GitHub

the tl;dr is that it’s a bug that manifests if your alert evaluation interval is 30s or more. It just got fixed in 8.1.7: Release notes for Grafana 8.1.7 | Grafana Labs

I was blocked by this for a long time until I found that issue thread. I hope this helps.

[libarton]

will give it a try, thank you!!