Grafana alerting error

hey daniellee, unfortunately due to security reason curl is not installed and i cant run an apt get command, but as you can see from the below image the datasource seems to be workig becoz all the graphs has been plotted.i’ve also made sure that the proxy settings are correct but i cant seem to understand why only the alerting feature seems to be giving errors.

hey daniellee, unfortunately due to security reason curl is not installed and i cant run an apt get command, but as you can see from the below image the datasource seems to be workig becoz all the graphs has been plotted.i’ve also made sure that the proxy settings are correct but i cant seem to understand why only the alerting feature seems to be giving errors.

You got it working with proxy mode on your machine now? What did you have to change to get it to work?

No, so the thing is this has been working for me since the beginning with direct mode itself, which is where im confused if there is a disconnect between influx and grafana how am i able to see the graphs and when i want to trigger alerts why am i getting err?
i just cant do the proxy mode , whenever i try to switch to proxy mode nothing works not even the graphs.

Direct mode means it goes directly from your computer to InfluxDB. So that means you have access to InfluxDB from your machine but your Grafana server does not. To get alerting to work, the Grafana server has to have access to InfluxDB.

I have no idea why that is. That is why I mentioned firewall settings before. It could be DNS problems. Can you even ping the InfluxDB from your Grafana server? (Or telnet if you don’t have ping).

Yes i’m able to ping InfluxDB from grafana server.

I don’t know the answer here. Have you got an Ops person that can help you check firewall rules, ports, proxy settings etc.

So yeah I’ve already set the proxy settings on my grafana and influxdb containers? Does that not suffice ?

Hey Danielle, ive done a few settings and the datasource is working with proxy mode but when i try to send the test alert it throws me the below errr
2017-05-04T16:19:53+0000 lvl=eror msg=“Failed to send slack notification” logger=alerting.notifier.slack error=“Post https://hooks.slack.com/services/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: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)” webhook=TestAlert
t=2017-05-04T16:19:53+0000 lvl=eror msg=“Failed to send alert notifications” logger=context userId=1 orgId=1 uname=admin error=“Post https://hooks.slack.com/services/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: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)”
t=2017-05-04T16:19:53+0000 lvl=eror msg=“Request Completed” logger=context userId=1 orgId=1 uname=admin method=POST path=/api/alert-notifications/test status=500 remote_addr=172.17.0.1 time_ms=30.002µs size=48

2. although when i check the test rule in the dashboard it works fine i guess? and then it throws me this error

t=2017-05-04T16:22:26+0000 lvl=info msg=“Sending notifications for” logger=alerting.notifier ruleId=1 sent count=1
t=2017-05-04T16:22:26+0000 lvl=info msg=Rendering logger=png-renderer path=“dashboard-solo/db/dashboard_04-14-2017?&panelId=1”
t=2017-05-04T16:22:27+0000 lvl=info msg=uploaded logger=alerting.notifier url=
t=2017-05-04T16:22:27+0000 lvl=info msg=“Sending notification” logger=alerting.notifier type=slack id=1 isDefault=true
t=2017-05-04T16:22:27+0000 lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=1 notification=TestAlert
t=2017-05-04T16:22:56+0000 lvl=eror msg=“Failed to send slack notification” logger=alerting.notifier.slack error=“context deadline exceeded” webhook=TestAlert
t=2017-05-04T16:23:01+0000 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalHandler ruleId=1 name=TestAlert changing state to=no_data

It’s timing out when trying to contact Slack. Either a proxy or SSL/TLS issue I’d guess. What proxy are you using for your Grafana server and what do the settings look like?

By the way, what was the problem with the proxy mode? How did you fix it?

first of all thanks for your prompt help everytime i needed, and the issue was with the proxy settings internal to our company. i just used the https and http proxies of our company which helped me out to run with the proxy.

Hi,
Thanks for all the details here.
But I am also getting the error “tsdb.HandleRequest() error Influxdb returned statuscode invalid status code: 400 Bad Request”.
I checked with grafana logs and nothing is getting logged for this error.
I checked the datasource, it is set for Proxy access.
I have the VM, grafana and influxdb is running on the same vm so assuming there is no firewall or access issue.

can you help ?

Hi Shafiq, the ERR said above is that you are getting when you are trying to send out an alert?

Hello ,

How you resolved the proxy issue . What settings need to be done on grafana for this .

I am facing the same issue.
I am using Grafana with Influxdb
The graphs are loading without any issue.
Grafana is running as a docker container without proxy settings.
When i test my alert rule, its throwing as
tsdb.HandleRequest() error Get : net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
Wondering is data for graph is working without any issues, but when i hit the test rule in alert tab its throwing the exception.

If i set the proxy for grafana server the alert test rule is also working.

But i need to run without any proxy settings.

Thanks in advance…