Hello,
I’ve been using Prometheus Alert Manager for ages, but I’m trying to move this to Grafana-managed alerts instead, however, it seems that using inline queries in annotations doesn’t seem to actually evaluate correctly and produces the incorrect output.
E.g. I have an alert such as:
- alert: "Server Unavailable"
for: 300s
expr: count:servers_unavailable >= 5
annotations:
summary: "Server {{ $labels.instance }} unavailable"
severity: danger
description: |
Server {{ $labels.dimension }} down for more than 5 minutes on 5 hosts or more:
{{ range printf `servers_unavailable_reason{dimension='%s'}` $labels.dimension | query }}{{ .Labels.instance }} {{ .Value }};{{ end }}
This works perfectly fine in Prometheus alert rules, it will execute the query servers_unavailable_reason with $labels.dimension as a parameter, and output it in a loop.
However, doing the same in Grafana’s UI in the Summary annotation, it simply yields nothing. Even if I simply output test in the for loop, it’s always empty.
I’ve tried various flavors based on Template examples | Prometheus range query, with query, with printf etc, and they all seem to produce nothing in Grafana, but works fine within native Prometheus alerts.
Is this expected behavior that we can’t seem to execute such inline queries in annotations in Grafana?