Alerting problem: error message "error="context deadline exceeded"

Hi
alerts notification does not work since I added a second channel (webhook). I have twho channels: this webhook and Pushover, I did not use Include image which seems to be a problem.

When a alert should be trigger, none of this two channels is sent. I have this error in log files:

t=2020-03-27T16:07:30+0000 lvl=info msg=“Sending webhook” logger=alerting.notifier.webhook
t=2020-03-27T16:08:00+0000 lvl=eror msg=“Failed to send webhook” logger=alerting.notifier.webhook error=“context deadline exceeded” webhook=“Notif Node-Red”
t=2020-03-27T16:08:00+0000 lvl=eror msg=“failed to send notification” logger=alerting.notifier uid=LjwrgbuWk error=“context deadline exceeded”
t=2020-03-27T16:08:00+0000 lvl=eror msg=“failed to send notification” logger=alerting.notifier uid=LjwrgbuWk error=“context deadline exceeded”
t=2020-03-27T16:08:00+0000 lvl=eror msg=“Failed to send pushover notification” logger=alerting.notifier.pushover error=“context deadline exceeded” webhook=Maison_High
t=2020-03-27T16:08:00+0000 lvl=eror msg=“failed to send notification” logger=alerting.notifier uid=ZGG0r__Zz error=“context deadline exceeded”
t=2020-03-27T16:08:00+0000 lvl=eror msg=“failed to send notification” logger=alerting.notifier uid=ZGG0r__Zz error=“context deadline exceeded”

Can you help me to resolve this problem?

context deadline exceeded means it is timing out when trying to send the notification.

If you only enable one, does it work then? Are you able to figure out which notification is causing the problem?

Yes, if I only enable one (Pushover notification) it works.
The problem is the webhook notification: it is a notification to Node-red with an HTTP Post. When I am clicking on Send Test button, it works but not when an alarm is raising.

Can you see any errors in the logs for Node-red?

You can also turn on debug logging for alert notifiers:

[log]
mode = console
filters = alerting.notifier:debug
level = info

I’m receiving an errors when enable any plugins when I’m running the service using the environment variable GF_INSTALL_PLUGINS:

Error: ✗ Failed to send request: Get “https://grafana.com/api/plugins/repo/grafana-clock-panel”: context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Without plugins service is running properly.

@Artur did you figure this one out? I face the same issue. It’s like the GF_INSTALL_PLUGINS use is not idempotent. Apparently it comes from this line: grafana/run.sh at ad63f871ce32a8982b34c9bf4be57fc593bb86aa · grafana/grafana · GitHub

I use it from systemd file like this:

[Unit]
Description=grafana Podman Container
After=network.target

[Service]
Type=simple
TimeoutStartSec=15
ExecStartPre=-/usr/bin/rm -f %T/%n-pid %T/%n-cid

ExecStart=/usr/bin/podman run --name grafana \
  --rm -p 3000:3000/tcp -e "GF_INSTALL_PLUGINS=flant-statusmap-panel" -v "/var/lib/containers/exported_volumes/grafana-etc/custom.ini:/etc/grafana/grafana.ini:Z" -v "/var/lib/containers/exported_volumes/grafana-lib:/var/lib/grafana:Z" --hostname="grafana" --label "io.containers.autoupdate=image" \
  --conmon-pidfile  %T/%n-pid --cidfile %T/%n-cid \
  docker.io/grafana/grafana 
ExecStop=/usr/bin/sh -c "/usr/bin/podman stop -t "15" `cat %T/%n-cid`"
ExecStop=/usr/bin/sh -c "/usr/bin/podman rm -f `cat %T/%n-cid`"
Restart=on-failure
RestartSec=30
KillMode=mixed
PIDFile=%T/%n-pid
Environment=PODMAN_SYSTEMD_UNIT=%n

[Install]
WantedBy=default.target

I issued the error here: Run grafana with docker without internet. Client.Timeout exceeded while awaiting headers · Issue #46460 · grafana/grafana · GitHub

Its solved here: Plugin installation fails without internet · Issue #60376 · grafana/grafana · GitHub

1 Like