I’ve set up Grafana (using Prometheus as a Datasource) to monitor one of my APIs. I then created an Alert Rule for this. After which I did a webhook integration to my SMS Webhook. The contact point and notification rules have been set up and associated with an ALERT Rule. When I test the connection to my using the “TEST” option the notification data is send successfully by Grafana to my webhook but when the ALERT Rule fires the Alert the notification Grafana doesn’t send the notification to my webhook (same URL and configuration as to the one that works when I use the “TEST” option - there aren’t any firewall rules in place that are preventing connection to where the webhook is running). I also successfully tested the connection to the webhook using Postman (Postman running on the same machine that Grafana is running from)
Note that I’m running Grafana locally (as a service) on a Windows Server 2022. I’m running Grafana v10.3.1 (00a22ff8b2).
I’ve increased the alert notification timeout in the configuration ini file from 30 to 60s (and 120s) but that didn’t make any difference
The error that i’m getting is “notify retry canceled due to unrecoverable error after 1 attempts: Post "https://smpp.resit.co.za/api2/grafsms\“: context deadline exceeded (Client.Timeout exceeded while awaiting headers)””
I’m sorry to hear that you’re experiencing difficulties.
Grafana can provide valuable information about certain issues through its own debug logs. These logs can help troubleshoot and identify the root cause of problems (especially if they are related to anything that can be configured via Grafana config file).
To enable debug logging edit the configuration file grafana.ini:
[log]
# Either "console", "file", "syslog". Default is console and file
# Use space to separate multiple modes, e.g. "console file"
mode = console file
# Either "debug", "info", "warn", "error", "critical", default is "info"
level = debug
Then restart grafana for the setting to go into effect.
Replicate the problem and check Grafana logs. However, before posting the debug logs on the community forum, it is important to sanitize any private details such as passwords, tokens, IPs, names, … By including properly formatted debug logs (with sanitized information) and details about your installations (e.g. exact Grafana version, how it was installed, OS details, …), you greatly enhance the chances of receiving accurate assistance and solutions.
HI @jangaraj
I switched on the debugs but still cant pinpoint what the issue is. Here is a snippet (NB: I mentioned the Grafana Version, OS etc in my previous comment when i asked for assistance from the community)
