Accessing Query Expression Results for Custom Notification Templates in Managed Grafana Alerts

Grafana Alerting
,
1

I’m currently facing an issue with accessing the results of a Query Expression within my custom notification template for Managed Grafana alerts. Despite configuring everything correctly (to my knowledge), when I try to use the template variable {{ .Values.value }} to display the query result value in my alarms, it consistently returns 0, indicating that the actual value is not being fetched or accessed correctly. Attached is a screenshot of the Metric expression setup for context.

image
image2466×1178 208 KB

I have already invested a considerable amount of time trying to troubleshoot and resolve this issue but to no avail. If anyone here has encountered a similar problem or has insights on how to properly access and use Query Expression results in custom notification templates for alerts in Managed Grafana, your guidance would be greatly appreciated. Below is a snippet of how I’m attempting to access the value in the template:

*Value*: {{ .Values.value }}

Thank you in advance for your help and suggestions!

2

so B value(s) will be {{ index $values "B" }} or {{ $values.B }}.

But this is not a task for notification template. You should use it in the alert annotation, e.g. meaningful description:

Pod: {{ $labels.container }} in namespace: {{ $labels.namespace }} on cluster: {{ $labels.kubernetes_cluster }} has high CPU utilization: {{ humanize $values.B.Value }}%

and notification template just use that annotation.

1 Like
3 
$values does not work

Tried that already and got an error.

Error: failed to save and apply Alertmanager configuration: template: slack:20: undefined variable “$values”

Also, this notification is not restricted to a single metric it is meant to work for multiple data sources like Cloudwatch, Prometheus, and Loki.

4

As I said: that’s for annotation, not for notification template.

1 Like
5

The best practice to use value (i.e. from section “B”) in notification template is:

  1. Edit your alert rule. Add this string to “Description” section in “Add annotations” part:
    {{ $values.B }}
  2. Then go to the Alerting->Contact Points->Notification Templates and add this string to your template:
    Value is: {{ .CommonAnnotations.description }}

After firing the alert you should see your value inside the alert report.
Good luck!

6

I marked the above reply as the solution.

You should use alert rule annotations such like the description to add extra info to the alert. This information can then be visible on the alert state history within Grafana and in notifications.

In the Template notification docs:

Note
Avoid adding extra information about alert instances in notification templates, as this information will only be visible in the notification message.
Instead, you should use annotations or labels to add information directly to the alert, ensuring it’s also visible in the alert state and alert history within Grafana. You can then print the new alert annotation or label in notification templates.