Using Grafana Cloud, I created an alert rule with annotations. This works well when alert notifications fire, but it doesn’t work when the alert is resolved. Labels work fine, so I’m wondering why annotations don’t?
I wrote the alert description like this:
{{ .Labels.invDeviceModel}} {{ .Labels.invDeviceName }} is missing
I use it in a notification template like this:
- {{ if .Annotations.description }}{{ .Annotations.description }}{{ end }}{{ if .Labels.space_name }} | {{ .Labels.space_name }}{{ end}}
When the alert fires, the notification text in Slack looks like this:
- TSC-80-G2 Epic-SL1-TSC is missing | Sound Lab-EPI-08.6.06
When the alert is resolved, the text in Slack looks like this:
This happens when the series for an alert instance disappears from the data source. When that occurs, the instance transitions to the Normal state as Resolved if it was firing.
From docs:
Stale alert instances in the Alerting, No Data, or Error states transition to the Normal state as Resolved, and are routed for notifications like other resolved alerts.
In this case, query labels will be missing.
This might explain your situation if invDeviceModel and invDeviceName are query labels, but space_name is a user-defined label (see about label types).
Thanks for the idea. I don’t think that applies here because all 3 of these labels come from the query. So if the series disappeared then space_name also wouldn’t work.
If the series disappears, the alert instance (including all its previous labels) remains active, but the labels from the missing series are set to empty values.
Do invDeviceModel, invDeviceName, and space_name come from the same query?
I’m asking because it’s possible that the query returning invDeviceModel and invDeviceName stopped producing data, while the space_name query is still returning results.
———————-
After a second view, I noticed the description annotation template might be wrong.
Yes those labels all come from the same query. The alert rule query is simply qsys_peripheral_status{allow_alerts="True"} == 4.
Thank you for pointing out the syntax difference. I updated my alert rule to use $labels instead of .Labels and now it’s working correctly for both the alert and the resolved message.
I also tested again with the wrong syntax, and found that it’s also working fine now! Weird…
