Annotations for alerts created with label variables

Grafana Alerting
,
1

Hello,

I am on a Mac OS using Grafana V12.1/ Prometheus and I am trying to show annotations on a panel for an alert I created in Grafana. I created this alert using label variables through its origin panel. I expected the annotations to show up on the panel relative to which label variable was selected but instead all the annotations of all alert instances show up at once on my panel. Is there any way to filter these annotations? The only filter available in the annotation section is through tags which I believe can only be manually implemented. Can tags be added with alerts? If so, the filter will be useful.

All the best

2

Hi @nicole5faa,

Trying to help here, but I’m not sure I fully understand your case. Anyway, I’ll share the way users usually work with alerts that are linked to dashboards using variables.

  1. We’d define a dashboard that implements a variable which can be filtered. For instance, filter dashboard results per service.

  2. An alert rule is defined to evaluate all services. This is implemented as a multi-dimensional alert rule. The alert rule could trigger as many alerts as there are existing services.

  3. You can connect the alert rule to a panel; see details in Create and link alert rules to panels.

  4. When an alert instance is triggered for a particular service, the alert notification should reference the dashboard panel filtered by that specific service. For this, the dashboard URL should include the service value to filter by that service.

This issue details how to do this.

In annotations:

dashboard?orgId=1&from=now-6h&to=now&var-cluster={{ $label.cluster }}

In notification templates:

dashboard?orgId=1&from=now-6h&to=now&var-cluster={{ .Labels.cluster }}

Let me know if I understood correctly and whether this helps!

3

Thank you for this reply! The issue was that we couldn’t get the multidimensional alert annotations to show up on the proper repeated panel. Since then we’re just now feeding the alerts through Prometheus and now we have full annotation capabilities.

Thank you for circling back though and I will keep this in mind for future projects :folded_hands:t2:

Best,
-Nicole