Grafana 4.6 Slack/Pagerduty Alert Notifications gives (Client.Timeout exceeded while awaiting headers)

1

Hi Grafana Users,

We are trying to setup and evaluate Grafana 4.6.0 on RHEL 7.0. One of the feature we are checking is Configuring Alert Notifications using Slack/PagerDuty. We have tried to configure an Alert on a Slack Webhook and when we tried to trigger, it looks like the Alerts are not being sent.

Slack - Grafanalog:

t=2018-02-20T21:20:00+0000 lvl=info msg=“Sending notification” logger=alerting.notifier type=slack id=0 isDefault=false
t=2018-02-20T21:20:00+0000 lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=0 notification=testslack
t=2018-02-20T21:20:30+0000 lvl=eror msg=“Failed to send slack notification” logger=alerting.notifier.slack error=“Post Unlock your productivity potential with Slack Platform | Slack net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)” webhook=testwebhook
t=2018-02-20T21:20:30+0000 lvl=eror msg=“Failed to send alert notifications” logger=context userId=** orgId=*** uname=**** error=“Post Unlock your productivity potential with Slack Platform | Slack net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)”
t=2018-02-20T21:20:30+0000 lvl=eror msg=“Request Completed” logger=context userId=* orgId=uname=* method=POST path=/api/alert-notifications/test status=500 remote_addr=... time_ms=30014 size=48 referer=https://.../alerting/notification/new

PagerDuty - Grafanalog:

t=2018-02-20T21:26:13+0000 lvl=info msg=“Sending notification” logger=alerting.notifier type=pagerduty id=0 isDefault=false
t=2018-02-20T21:26:13+0000 lvl=info msg=“Notifying Pagerduty” logger=alerting.notifier.pagerduty event_type=trigger
t=2018-02-20T21:26:43+0000 lvl=eror msg=“Failed to send notification to Pagerduty” logger=alerting.notifier.pagerduty error=“Post https://events.pagerduty.com/generic/2010-04-15/create_event.json: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)” body=“{"client":"Grafana","client_url":"http://localhost:8000/\",\“contexts\”:[{\“src\”:\“http://grafana.org/assets/img/blog/mixed_styles.png\”,\“type\”:\“image\”}],\“description\”:\"Test notification - Someone is testing the alert notification within grafana.","details":"Triggered metrics:\n\nHigh value: 100.000\nHigher Value: 200.000\n","event_type":"trigger","incident_key":"alertId-0","service_key":"*******"}”

Can any one advice me what could be wrong with the errors shown from grafana.log and if there is anything special we would need to do if we are running behind proxy?

Thank you

2

Hi,

Do you happen to have a firewall blocking those calls?

Marcus

3

Thank you @mefraimsson for replying back. I was mentioned our enterprise proxy has the connectivity opened when i asked about it and can ask back about the firewall to them. Is there a way I could verify if that connectivity through any small test from grafana server. I was also curios if a proxy end point(http://user:accnt@proxy..com:***)can be configured in grafana.ini to make sure the requests are tested.

Thank you

4

Well, you could try and use ping and/or curl from the server where grafana is installed and see if you get proper respones, i.e.

ping hooks.slack.com
curl https://events.pagerduty.com/generic/2010-04-15/create_event.json

Also, make sure that the user account that grafana runs under have access to do external request.

Marcus

5

Hi @mefraimsson,

My apologies for the delayed response. I have tried to perform the above operations after configuring proxy env variables in /etc/sysconfig/grafana-server and restarted my grafana-server. Please find in my observations. I wasn’t sure what else i could verify to test the connectivity.

Slack:
ping hooks.slack.com
PING hooks.slack.com (x.y.abc.def) 56(84) bytes of data.
^C
— hooks.slack.com ping statistics —
5 packets transmitted, 0 received, 100% packet loss, time 3999ms

pagerduty:
curl https://events.pagerduty.com/generic/2010-04-15/create_event.json
{“status”:“invalid method”,“message”:“You must use HTTP POST to submit your event”}

Thank you.

6

Hi,

You cannot reach hooks.slack.com. Seems like something is blocking those requests. Firewall somewhere maybe?

Marcud

7

Hi @mefraimsson,

I was mentioned that firewall config was open through proxy for accessing slack. I will try setting proxy and give this a try. Thank you