Grafana won't fully render Slack template

Grafana Alerting
,
1

I have Grafana alerting to Slack and now I want to add additional annotation: traces to be displayed in the alert. For some reason it is not being shown in the alert, but I can see this new annotation in Grafana alert in the UI.

My Grafana Slack template:

{{ define "alert_severity_prefix_emoji" -}}
        {{- if ne .Status "firing" -}}
                :white_check_mark:
        {{- else if eq .CommonLabels.severity "critical" -}}
                :red_circle:
        {{- else if eq .CommonLabels.severity "warning" -}}
                :warning:
        {{- end -}}
{{- end -}}

{{ define "slack.title" -}}
  {{ template "alert_severity_prefix_emoji" . }}
        [{{- .Status | toUpper -}}{{- if eq .Status "firing" }} x {{ .Alerts.Firing | len -}}{{- end }}]: {{ .CommonLabels.alertname -}}
{{- end -}}

{{- define "slack.text" -}}
{{- range .Alerts -}}
{{ if gt (len .Annotations ) 0 }}
*Description*: {{ .Annotations.description }}
*Instance*:
{{ range .Labels.SortedPairs }}{{ if or (eq .Name "env") (eq .Name "instance") (eq .Name "cluster") (eq .Name "http_host")}}• {{ .Name }}: `{{ .Value }}`
{{ end }}{{ end }}
*Silence alert*: {{ .SilenceURL }}
*Go to dashboard*: {{ .DashboardURL }}
*Go to panel directly*: {{ .PanelURL }}
*Go to traces*: {{ .Annotations.traces }}
{{ end }}
{{ end }}
{{ end }}

In the alert rule I have:

resource "grafana_rule_group" "prometheus_metrics_rule_group" {
  name             = "Prometheus metrics"
  folder_uid       = grafana_folder.prometheus_rule_folder.uid
  interval_seconds = 300
  org_id           = 1
  rule {
    name           = "5xx HTTP server side errors"
    for            = "2m"
    condition      = "B"
    no_data_state  = "OK"
    exec_err_state = "Alerting"
    annotations = {
      __dashboardUid__ = grafana_dashboard.prometheus_http_errors.uid
      __panelId__      = "1"
      "description" = "{{ with $values }} 5xx server side HTTP errors have exceeded 10% of all requests within the last 10 minutes"
      "traces" = "{{ graphLink \"{\"expr\": \"up\", \"datasource\": \"Tempo\"}\" }}"

For the test purposes I’ve tried set “traces” annotation to just a “test” string but with no luck. What I am doing wrong here?

1 Like
2

It works for me :confused: I’m testing it in the Preview feature in Grafana 10

Screenshot 2023-09-05 at 8.42.26 pm
Screenshot 2023-09-05 at 8.42.26 pm1236×954 72.2 KB

3

It works for me in preview too, but won’t show in Slack message. Is there any limit for message length?

4

So during my trial and error tests I found that replacing/moving:

*Go to traces*: {{ .Annotations.traces }}

with:

*Silence alert*: {{ .SilenceURL }}

worked and now I can see it in the Slack alert message. I suspected some kind of character limit in single Slack message but it is not that. I would say this is a bug except there is another explanation for this.