Unable to send notifications to mattermost

I was trying to send alert notifications to mattermost. From what I understand, it uses a webhook similar to Slack.

However, when I configure it, the test notification fails with the following error in the system logs

==> /var/log/messages <==
Jul 6 16:52:49 myservername grafana-server: t=2017-07-06T16:52:49+1200 lvl=dbug msg=“Scheduling update” logger=alerting.scheduler ruleCount=0
Jul 6 16:52:51 myservername grafana-server: t=2017-07-06T16:52:51+1200 lvl=info msg=“Sending notification” logger=alerting.notifier type=slack id=0 isDefault=false
Jul 6 16:52:51 myservername grafana-server: t=2017-07-06T16:52:51+1200 lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=0 notification=MatterMost
Jul 6 16:52:51 myservername grafana-server: t=2017-07-06T16:52:51+1200 lvl=dbug msg=“Sending webhook” logger=notifications.webhook url=https://mattermostservername-front.elinux.abc.co.nz/hooks/8974yo4o37nyfdxjwooduh http method=
Jul 6 16:52:51 myservername grafana-server: t=2017-07-06T16:52:51+1200 lvl=dbug msg=“Webhook failed” logger=notifications.webhook statuscode=“500 Internal Server Error” body=
Jul 6 16:52:51 myservername grafana-server: t=2017-07-06T16:52:51+1200 lvl=eror msg=“Failed to send slack notification” logger=alerting.notifier.slack error=“Webhook response status 500 Internal Server Error” webhook=MatterMost
Jul 6 16:52:51 myservername grafana-server: t=2017-07-06T16:52:51+1200 lvl=eror msg=“Failed to send alert notifications” logger=context userId=1 orgId=1 uname=admin error=“Webhook response status 500 Internal Server Error”
Jul 6 16:52:51 myservername grafana-server: t=2017-07-06T16:52:51+1200 lvl=eror msg=“Request Completed” logger=context userId=1 orgId=1 uname=admin method=POST path=/api/alert-notifications/test status=500 remote_addr=127.0.0.1 time_ms=19ns size=48

I tried to do a curl command on the same linux box where grafana is installed, and the command succeeds.

Where could I be going wrong?

I guess that the Mattermost API is not 100% compatible with the Slack API. You are getting a HTTP 500 response which indicates an error.

Here is the JSON document that gets sent to the Slack hook:

Thanks Daniel!

I also tried with the ‘webhook’ option with the same result. Wondering if there is a workaround which we could use to get this working.

With the webhook option you will have to do some programming and build your own service that can talk to mattermost.

Hi,
this works for us.

Our versions are :

  • Grafana 5.2.4
  • Mattermost 5.2.1

Hope this helps

1 Like

hi bekir
Could you elaborate on how you got it working?

  • We have Mattermost 5.7.0 and using the hosted cloud version of Grafana.
  • The webhook works (we can post via curl)
  • Setting up a webhook notification yields no errors in Grafana, but nothing happens in Mattermost

Thanks

@erikohlzon Hi Erik,

I’m currently having the same problem as you did. Were you able to solve it?

Thanks

No, it still does not work. Seams to be the way Grafana sends the message is not in line with the standard MM expects

Hey Bekir,

I am also trying to achieve the same thing, but I couldn’t find any resources how to do this integration for mattermost. Soi it would be very useful, if you could provide the details how you’ve configured the alert channel in grafana.

I’ve seen this post of yours -> https://forum.mattermost.org/t/mattermost-grafana-integration/5945

is this how you’ve configured your notification channel in grafana?

Thanks in advance!

Hi,

Yes, that’s how I configured it. And apart from the image problem, it worked properly.

(Unfortunately, I won’t be able to help you more on this issue as I don’t have access to that setup anymore : I left the company).

Hope this helps