Background information:
We have alerts configured via Prometheus Alertmanager and both Prometheus and Prometheus Alertmanagers are configured as datasources for Grafana.
The challenge:
How to show annotations in Grafana panels when alert is triggered?
Ideally the generated annotation would be range annotation (start time when it was triggered and end time when it was resolved).
Ideally the Grafana annotation created when alert is triggered would get tags based on alert definition (i.e. k8s cluster, namespace, container, …), so it could be rendered in all panels of Grafana dashboards, that have Annotation Query configured based on tags and selected variable values matching tags of the triggered Annotaion (i.e. using Annotation Query “tag: cluster:$cluster and namespace:$namespace” that matches Annotations that have both “cluster:staging” and “namespace:demo” tags when Grafana dashboard variables are set to cluster=staging and namespace=demo)