Alert not working

[thomasthomas]

Version: Grafana 7.3.7
OS: Ubuntu 18.04
Issue: Alert not working after the first evaluation
Expected behaviour: Alert query is supposed to be evaluated each hour.
Data source: Prometheus 2.21.0
Error message: no message in the UI and in the grafana-server.log

If I test the alert rule manually, it works. Evaluate every is set to 1h and For is set to 0m.

I tested whether reducing the scheduling from 1h to 1m has any effect - but after the first evaluation everything seemed to be pretty much the same.

Here’s the output that I get if I test the rule manually:

Object

firing:false

state:"ok"

conditionEvals:"false = false"

timeMs:"35.626ms"

logs:Array[3]

0:Object

message:"Condition[0]: Query"

data:Object

1:Object

message:"Condition[0]: Query Result"

data:Object

2:Object

message:"Condition[0]: Eval: false, Metric: disk utilisation, Value: 0.054"

data:null

Any idea?

[aparajitachatterjee]

Hello,

Here is a troubleshooting topic for alerts: http://localhost:3002/docs/grafana/latest/alerting/troubleshoot-alerts/
Let us know if this helps. Thank you.

[thomasthomas]

Thank you for your response, @aparajitachatterjee .

Before I take a deep dive into such low-level logging, I want to be certain when there are multiple status records inside the status history in the Alert tab.

Previously I thought something had been wrong as my pre-defined query was supposed to be evaluated each hour, however after the first run [that returned OK status] it did not appear in the history tab.

However, now I think it is ok - it would not be informative to see the evaluated query each hour if it is already visible in the graph. Instead, the history should contain the timestamps when there was a change in alert status.

So the question is this: where can I check the last time when the alert query was run?

[willbrowne]

Hi @thomasthomas,

Instead, the history should contain the timestamps when there was a change in alert status.

This should be shown in the State History tab on the alert view page. It should look something like

This information should also be viewable from the panel itself in the form of annotations.

where can I check the last time when the alert query was run?

This will be run according to the evaluation period set on the alert. But AFAIK we don’t expose that information for every run, only the runs where there are state changes