Alert: firing, notification: tested, combination of both: broken

  • What Grafana version are you using?
    4.6.1

  • What datasource are you using?
    Graphite 0.9.x via proxy

  • What OS are you running grafana on?
    DC/OS

  • What did you do?
    I set up an alert and set it to send a notification to a slack channel

  • What was the expected result?
    When the rule fires, the notification should arrive in the respective slack channel

  • What happened instead?
    No notification. In the grafana log, I see:
    t=2017-11-08T09:01:07+0000 lvl=eror msg=“Failed to send slack notification” logger=alerting.notifier.slack error=“context deadline exceeded” webhook=“Availability Alert”

The alert is firing alright, which I see in the graph and also when clicking on “test rule”.
The notification is generally also working: the test notification message arrives every time in the slack channel. From other issues I gathered that “context deadline exceeded” hints at connectivity issues but since the test and other notifications arrive in other slack channels ok, connectivity with slack doesn’t seem to be the issue.

I have recreated the alert from scratch and deleted the history but it didn’t help.

strange that test notification channel works but not the actual notification. Some connection and timeout issue. We have no other reports on this error in the slack notifier so no idea what can cause it.

any idea on how to proceed?