I’ve recently come back to Grafana (now backed by Prometheus) after a long time away from the project and am working to set up some basic alerting, using 9.3.2 with Unified Alerting. I’ve got it working great for “simple” alert rules created from a panel, and I have panel images coming through in email alerts via the grafana-image-renderer sidecar. However, I have two cases that are really confusing me and I can’t seem to figure out:
- Assuming I have a multi-valued query in my panel, an alert rule created from the panel (and linked to it) works fine, but the panel image included in the alert is the literal panel with all values, both alerting and not. Is there any way around this, aside from creating a separate panel for each value/series? And if not, I’m assuming there’s not a way to… dynamically associate an alert with a panel? Or associate alert instances (instead of rules) with panels?
- Is there any way to associate alert rules with a panel in a “repeat for” row on a dashboard? And if so, can this be done dynamically?
Essentially my use case boils down to two main requirements:
- Every alert message (at least email) should include a corresponding graph/panel snapshot for context.
- Alerts (and the corresponding panel) should be created dynamically, i.e. the way “Repeat for” works on a dashboard.
My research over the past few days seems to indicate that the only way I can get the behavior I want is to write something that creates/manages alerts and a corresponding panel for every individual alert, and then regularly pulls data from the data source (Prometheus) to add new alerts and panels as needed. This feels like a bit of a pain, but more importantly feels like it’s really misusing Unified Alerting’s data model, if I’m creating a whole bunch of Alert Rules all of which have only one Instance.
Thanks for any advice.