Grafana webhook "Failed to send alert notifications" x509: certificate signed by unknown authority

Grafana
1

I have set up an Alert Notification type webhook and in webhook settings i provided the https url and using post method, but when i click on send test it says “failed to send notifications”.

When i checked the grafana logs in the server i can see an error message lvl=eror msg=“Failed to send alert notifications” logger=context userId=13 orgId=1 uname=****** error=“Post https://****************************: x509: certificate signed by unknown authority”.

I have added the cert.pem of the destination server to the local java keystore and i still get the same error.
Let me know if this is something to with adding certs of the destination server into the grafana.

Thank you

2

Hi,

No idea what you mean by this, but the backend in Grafana is written in golang. Guess you’ll need to put the cert somewhere on the computer where golang can find it, but not sure.

Marcus

3

Thanks for the quick response.
When i meant local keystore, keeping cert.pem inside /usr/java/jre/lib/security/cacerts, which worked when two servers were talking to each other over https.
But in this case where Grafana trying to send alert notification via webhook to an https client is failing with an error message x509: certificate signed by unknown authority.

image
image1341×542 76.9 KB

4

Not sure I can help you. Please find the default path on your system to store certificates and add the cert there. The Java path must be specific for Java applications so that can be correct.

5

Does Grafana supports self signed certificates for sending alerts via webhook to an https endpoint.

6

No. Thought that the operating system was somehow involved in solving this together with go but I was wrong.

Please refer to this feature request issue and upvote it with a :+1: emoji if you really want this feature. Contributions are always welcome if you think you can solve it yourself.

Marcus

7

you will have to follow
tail -f /var/log/grafana/grafana.log
your mailrc client should have the cert information populated along with your smtp information
certfile and key file needs to be added to the smtp section
,make sure they are chmod 0440 and that they have the appropriate cert chain on the cert file

append them all on one file your ca cert , and server cert

8

forghot to add skip-verify option make it true