Alerts with images - how to handle multivalue, repeat for rows, etc

Grafana Alerting
1

I’ve recently come back to Grafana (now backed by Prometheus) after a long time away from the project and am working to set up some basic alerting, using 9.3.2 with Unified Alerting. I’ve got it working great for “simple” alert rules created from a panel, and I have panel images coming through in email alerts via the grafana-image-renderer sidecar. However, I have two cases that are really confusing me and I can’t seem to figure out:

  1. Assuming I have a multi-valued query in my panel, an alert rule created from the panel (and linked to it) works fine, but the panel image included in the alert is the literal panel with all values, both alerting and not. Is there any way around this, aside from creating a separate panel for each value/series? And if not, I’m assuming there’s not a way to… dynamically associate an alert with a panel? Or associate alert instances (instead of rules) with panels?
  2. Is there any way to associate alert rules with a panel in a “repeat for” row on a dashboard? And if so, can this be done dynamically?

Essentially my use case boils down to two main requirements:

  1. Every alert message (at least email) should include a corresponding graph/panel snapshot for context.
  2. Alerts (and the corresponding panel) should be created dynamically, i.e. the way “Repeat for” works on a dashboard.

My research over the past few days seems to indicate that the only way I can get the behavior I want is to write something that creates/manages alerts and a corresponding panel for every individual alert, and then regularly pulls data from the data source (Prometheus) to add new alerts and panels as needed. This feels like a bit of a pain, but more importantly feels like it’s really misusing Unified Alerting’s data model, if I’m creating a whole bunch of Alert Rules all of which have only one Instance.

Thanks for any advice.

2

Hi!

There are two issues here:

  1. I don’t think we can show just one series in a panel without also changing the query
  2. Taking screenshots is very resource intensive, so alerts share the same screenshot if fired or resolved within the same minute

How many series do you have in the panel?

3

Ok, that’s all good to know, thanks!

  1. Taking screenshots is very resource intensive, so alerts share the same screenshot if fired or resolved within the same minute

I assume you’re talking about alerts that reference the same panel?

How many series do you have in the panel?

That differs. On average for cases like this, probably 4-6, but some have maybe up to 20 or so.

I’m pretty sure that, after some more experimentation over the last week, I’m going to resort to using something outside of Grafana to provision alerts and matching panels (on a “hidden” / otherwise unused dashboard) via the Provisioning APIs, to end up with what I want. It seems like this will be necessary for both dealing with multi-valued queries, as well as for dealing with alert images for dashboards that currently use “repeat for” and/or variables.

4

I assume you’re talking about alerts that reference the same panel?

Yes, correct!