Only one of two alerts are fired

Grafana
#1

Hi all,

I have setup a dashboard with multiple panels. Two of these panels have an alert, but only one of them is fired. The first panel is a query on a SQL Server, if this reaches a certain value an alert is raised and send to Slack and Pushover. This all works fine on this panel.

The second panel is checking if my service is running, using Promotheus and this query:

wmi_service_state{name=“transcontm”,state=“running”, instance=~“meg:9182”}

My alert is setup like this:

image
image753×356 15.6 KB

My panel is going into “alert mode” if the service is stopped, but no message is sent to Slack and Pushover. What am I doing wrong?

image

#2

Did you check Grafana logs for errors?
Did you set an alerting channel for this alert? (cant see in scrnr)

#3

The channel is setup and the first panel is sending alerts to this. But thank you, I found a problem about the second panel in the log files.

I seen an error “Failed to upload alert panel image.”. I switched off the “Include image” in the notification channel setting, after that the alert for both panels are sent OK.

Is it not possible to send an image from a Promotheus datasource, or is this a problem in my “query”?

t=2019-12-12T10:29:29+0100 lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=0 notification=Slack
t=2019-12-12T10:29:58+0100 lvl=info msg=“Rendering timed out” logger=rendering
t=2019-12-12T10:29:58+0100 lvl=eror msg=“Failed to upload alert panel image.” logger=alerting.notifier error=“Timeout error. You can set timeout in seconds with &timeout url parameter”
t=2019-12-12T10:29:58+0100 lvl=eror msg=“Failed to send pushover notification” logger=alerting.notifier.pushover error=“context deadline exceeded” webhook=Pushover
t=2019-12-12T10:29:58+0100 lvl=eror msg=“failed to send notification” logger=alerting.notifier uid=gRnQ0-aZz error=“context deadline exceeded”
t=2019-12-12T10:29:58+0100 lvl=eror msg=“failed to send notification” logger=alerting.notifier uid=gRnQ0-aZz error=“context deadline exceeded”

#4

Glad you found it, maybe it will help to activate debug logging?
Dont know if the plugin will write all the necessary informations into grafana log.

Grafana Log configuration