Alerting variables misbehaving (Not expanding {{ <variable> }}

Grafana Alerting
1

  • What Grafana version and what operating system are you using?

    • Grafana v12.3.1 (3a1c80ca7c) docker container on Ubuntu 24.04.3 LTS

  • What are you trying to achieve?

    • I am provisioning alerts using Yaml files

  • How are you trying to achieve it?

    • Provisioned contact points, Notification Templates, Notification policies, and alerting rules

  • What happened?

    • Everything is created successfully, but I have multiple issues:

      • In the Grafana interface, I get every variable presented as the variable, not as the expanded value (attaching an example):

        image
        image1852Ă—777 144 KB

      • The same alerts that I have provisioned elsewhere, are showing the variables, without expanding it’s values:

        image
        image646Ă—425 35.1 KB

  • What did you expect to happen?

    • I expect that the variables like $labels.fqdn, and $values.A.value are expanded from the query.

  • Can you copy/paste the configuration(s) that you are having problems with?

    • This is my notification template: {{ define “samm_summary” }}

       {{ if gt (len .Alerts.Firing) 0 }} 
    {{ len .Alerts.Firing }} firing: 
    {{ range .Alerts.Firing }} 
    {{ .Annotations.summary }} 
    {{ end }} 
  {{ end }} 
  {{ if gt (len .Alerts.Resolved) 0 }} 
    {{ len .Alerts.Resolved }} resolved: 
    {{ range .Alerts.Resolved }} 
    {{ .Annotations.resolved_summary }} 
    {{ end }} 
  {{ end }} 
{{ end }}

    • This is one of the notifications summaries that is not expanding variables:Uptime for multi-session VDA {{ $labels.fqdn }} is above 2 days ({{ $values.A.value }} days)

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

    • This is an example of the errors I’m getting 
      • grafana-1  | logger=ngalert.state.manager rule_uid=df7vzh8b26uwwe org_id=1 t=2025-12-22T18:18:00.027762229Z level=error msg="Error in expanding template" error="failed to expand template '{{- $labels := .Labels -}}{{- $values := .Values -}}{{- $value := .Value -}}Uptime for multi-session VDA {{ $labels.fqdn }} is above 2 days ({{ $values.A.value }} days) - alert resolved': error executing template __alert_test-uptime2days: template: __alert_test-uptime2days:1:151: executing \"__alert_test-uptime2days\" at <$values.A.value>: can't evaluate field value in type template.Value"
grafana-1  | logger=ngalert.state.manager rule_uid=df7vzh8b26uwwe org_id=1 t=2025-12-22T18:18:00.027874067Z level=error msg="Error in expanding template" error="failed to expand template '{{- $labels := .Labels -}}{{- $values := .Values -}}{{- $value := .Value -}}Uptime for multi-session VDA {{ $labels.fqdn }} is above 2 days ({{ $values.A.value }} days) - alert resolved': error executing template __alert_test-uptime2days: template: __alert_test-uptime2days:1:151: executing \"__alert_test-uptime2days\" at <$values.A.value>: can't evaluate field value in type template.Value"
grafana-1  | logger=ngalert.state.manager rule_uid=df7vzh8b26uwwe org_id=1 t=2025-12-22T18:18:00.02795754Z level=error msg="Error in expanding template" error="failed to expand template '{{- $labels := .Labels -}}{{- $values := .Values -}}{{- $value := .Value -}}Uptime for multi-session VDA {{ $labels.fqdn }} is above 2 days ({{ $values.A.value }} days)': error executing template __alert_test-uptime2days: template: __alert_test-uptime2days:1:151: executing \"__alert_test-uptime2days\" at <$values.A.value>: can't evaluate field value in type template.Value"
grafana-1  | logger=ngalert.state.manager rule_uid=df7vzh8b26uwwe org_id=1 t=2025-12-22T18:18:00.02821775Z level=error msg="Error in expanding template" error="failed to expand template '{{- $labels := .Labels -}}{{- $values := .Values -}}{{- $value := .Value -}}Uptime for multi-session VDA {{ $labels.fqdn }} is above 2 days ({{ $values.A.value }} days) - alert resolved': error executing template __alert_test-uptime2days: template: __alert_test-uptime2days:1:151: executing \"__alert_test-uptime2days\" at <$values.A.value>: can't evaluate field value in type template.Value"
grafana-1  | logger=ngalert.state.manager rule_uid=df7vzh8b26uwwe org_id=1 t=2025-12-22T18:18:00.028387107Z level=error msg="Error in expanding template" error="failed to expand template '{{- $labels := .Labels -}}{{- $values := .Values -}}{{- $value := .Value -}}Uptime for multi-session VDA {{ $labels.fqdn }} is above 2 days ({{ $values.A.value }} days)': error executing template __alert_test-uptime2days: template: __alert_test-uptime2days:1:151: executing \"__alert_test-uptime2days\" at <$values.A.value>: can't evaluate field value in type template.Value"
grafana-1  | logger=ngalert.state.manager rule_uid=df7vzh8b26uwwe org_id=1 t=2025-12-22T18:18:00.028622692Z level=error msg="Error in expanding template" error="failed to expand template '{{- $labels := .Labels -}}{{- $values := .Values -}}{{- $value := .Value -}}Uptime for multi-session VDA {{ $labels.fqdn }} is above 2 days ({{ $values.A.value }} days) - alert resolved': error executing template __alert_test-uptime2days: template: __alert_test-uptime2days:1:151: executing \"__alert_test-uptime2days\" at <$values.A.value>: can't evaluate field value in type template.Value"

  • Did you follow any online instructions? If so, what is the URL?

I also recreated one of the same alerts manually, and the error persists, so it seems it is not related to provisioning, but I am at a loss on what can cause this. I deleted the Grafana container and image and recreated it also.

2

I have found something new. If the notification does not call a “value”(e.g. $values.A.value), the notification works as expected.

3

Found the error. it only works if values has a capital V: $value.A.Value, instead of $value.A.value. Devil is in the details!

1 Like