How to tell if triggered is by execution error or timeout?


How do I tell if the alert on Grafana Cloud is triggering due to execution error or timeout?

When there is no data, the alert state is set to OK. According to the alert history, the alert state does not transition to pending when going from Normal to Alerting. Looking at the data during the period that the alert was triggering, the data is outside the alert range. There are no labels on the alert in question, when there are if state does transition through Pending to Alerting. Therefore I suspect that the alert is being triggered by execution error or timeout, but I can’t see a way to identify this explicitly.

The alert is node_filesystem_avail_bytes{fstype!~"tmpfs|fuse.lxcfs"} < 1024 * 1024 * 200, which is then reduced using the function Last in strict mode. Therefore I don’t know why this would have an execution error or timeout.

Thank you

It seems that it is due to it failing to execute the query. I have changed the rule to have it error if there in an execution error, instead of alert. Hopefully that will make a difference in the alert.

1 Like

Changing to error provided the visibility I needed