Custom Labels Not Showing in Alert Rule Annotations in Grafana

Grafana
,
1

Grafana Version : 10.4
Hello Grafana Community,

I have recently added new alert rules in Grafana using the non-classic condition type for expressions, and tried to use custom labels in the annotations’ descriptions. However, the labels are not being displayed regardless of the methods I use. Here are the details of my issue:
Label Definition:
error = NOT_FOUND

Annotations in Alert Rule
Description:

  • This is error : {{ index $labels “error” }} → Displays as nothing
  • This is error : {{ $labels.error }} → Shows as [no value]
  • This is error : {{ .Labels.error }} → Shows as [no value]

Interestingly, when using the default template to display labels, the ‘error’ label is shown, indicating that the label is correctly added.

My goal is to have the custom label ‘error’ displayed in the description field of annotations when sending messages through Slack. Is this not possible?

Additionally, when I set {{ $labels }} as the description to test what gets sent, only the following four types of labels are displayed:

  • __alert_rule_namespace_uid__
  • __alert_rule_uid__
  • alertname
  • grafana_folder

Could anyone please help me understand why my custom label ‘error’ isn’t being displayed in the annotations and how I might fix this, especially considering I am using a non-classic condition expression? Any insights or suggestions would be greatly appreciated.

Thank you!

2

It is not clear how your alert query and alert preview looks like.

1 Like
3

1 (2)
1 (2)1369×1735 212 KB

I am using the alert query in this way.
Additionally, I have set up the labels and annotations as follows.
2 (2)
2 (2)1384×1355 76.4 KB

However, the actual content I receive via Slack messages is as follows:

[FIRING:1, RESOLVED:0] (ES)
Firing
Value: B=0, C=10
Labels:

  • alertname = test
  • error = NOT_FOUND
  • grafana_folder = ES
    Annotations:
  • description = alert_rule_namespace_uid=@@@@, alert_rule_uid=@@@@, alertname=test, grafana_folder=ES

The labels displayed in the message include the ones I’ve added, but they do not appear in the annotation description.

4

You don’t have any labels in the result from your query => so [no data] on the annotation level is correct:

image
image762×258 35.3 KB

You must see that error label in the preview first. Try to add grouping by error (msg) field and check preview.

1 Like
5

Thank you for your response.
I believe the information you provided will be helpful in solving the problem we are facing.

However, I still have a question: does the annotation only handle labels for queries? It seems that the scope of the labels might differ, and I would appreciate it if you could share any documents where I can learn more about this.

Thank you very much for your prompt reply and for helping to address the issue. :grinning:

1 Like
6

Yes, annotations work with labels from alert query. You can define other labels on the alert query - they will be appended to labels from query and listed/processed by notification template. It doesn’t make sense to create additional label on the alert level and then use it immediately in the annotation (you can write the same label value directly there - you don’t need to reference it with some label variable - so I think that’s the reason why devs don’t expand additional alerts labels there - only labels from the alert query).

1 Like
7

Your answer has been very helpful. Thank you for informing me about something I didn’t know. Have a great day :slight_smile: