Grafana Alerts Provisioning not working

[cruxforum]

I am trying to provision grafana alerts for which I am trying to pass all the data for groups, contactpoints, policies and templates via configmap. But I am facing issues while trying to deploy it. Please find the below configmap I am trying to create:

apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-alerts-configmap
namespace: istio-system
data:
alerting.yaml: |
apiVersion: 1
groups:
- orgId: 1
name: crux_alerts_group
folder: crux_alerts
interval: 60s
rules:
- uid: rule_1
title: Storage Volume Alert
condition: A
data:
- refId: A
datasourceUid: "-100"
model:
editorMode: code
expr: 'sum by (persistentvolumeclaim,namespace) (kubelet_volume_stats_used_bytes{namespace=<del>"".*""}/kubelet_volume_stats_capacity_bytes{namespace=</del>"".*""} * 100)'
hide: false
intervalMs: 1000
legendFormat: '{{persistentvolumeclaim}} : {{namespace}}'
maxDataPoints: 43200
range: true
refId: A
- refId: B
datasourceUid: "-100"
model:
conditions:
- evaluator:
params:
- 3
type: gt
operator:
type: and
query:
params:
- A
reducer:
type: last
type: query
datasource:
type: **expr**
uid: "-100"
expression: A
intervalMs: 1000
maxDataPoints: 43200
refId: B
type: reduce
- refId: C
datasourceUid: "-100"
model:
conditions:
- evaluator:
params:
- 3
type: gt
operator:
type: and
query:
params:
- B
reducer:
type: avg
type: query
datasource:
type: **expr**
uid: "-100"
expression: $B \u003e 70
intervalMs: 1000
maxDataPoints: 43200
refId: C
type: Math
dashboardUid: peR80gTGk
panelId: 29
noDataState: NoData
executionErrorState: Error
for: 60s
annotations:
summary: Alerting for High Memory Usage > 70% !!!
description: PVC Usage is {{ humanize $values.B.Value }} % on {{ $labels.persistentvolumeclaim }}.
labels:
team: crux

 contactPoints:
   - orgId: 1
     name: 'Crux Slack'
     receivers:
     - uid: cp_1
       type: slack
       settings:
         url: <url>
         icon_emoji: warning
         title: |
           {{ template "crux-slack.title" . }}
         text: |
           {{ template "crux-slack.text" . }}

 policies:
   - orgId: 1
     receiver: Crux Slack
     group_by:
     - grafana_folder
     - alertname
     matchers:
     - team = crux
     #- alertname = Watchdog
     #- severity =~ "warning|critical"
     mute_time_intervals:
     - abc
     group_wait: 30s
     group_internval: 5m
     repeat_interval: 4h
     routes:
     ...

 templates:
   - orgID: 1
     name: crux-slack
     template: |
       {{ define "alert_severity_prefix_emoji" -}}
       	{{- if ne .Status "alerting" -}}
       		:warning:
       	{{- end -}}
       {{- end -}}

       {{ define "crux-slack.title" -}}
       	{{ template "alert_severity_prefix_emoji" . }}
       	[{{- if eq .Status "firing" }} x {{ .Alerts.Firing | len -}}{{- end }} ] ||  {{ .CommonLabels.alertname -}}
       {{- end -}}

       {{- define "crux-slack.text" -}}
       {{- range .Alerts -}}
       {{ if gt (len .Annotations) 0 }}
       *Environment*: *GCP Dev*
       *Summary*: {{ .Annotations.summary}}
       *Description*: {{ .Annotations.description }}
       *DashboardURL*: {{ if gt (len .DashboardURL ) 0 }}
          {{ .DashboardURL }}{{ end }}
       *PanelURL*: {{ if gt (len .PanelURL ) 0 }}
          {{ .PanelURL }}{{ end }}
       *Labels*:
       {{ range .Labels.SortedPairs }}{{ if or (eq .Name "container") (eq .Name "pod") (eq .Name "persistentvolumeclaim") (eq .Name "node") (eq .Name "namespace") }}• {{ .Name }}: `{{ .Value }}`
       {{ end }}{{ end }}
       {{ end }}
       {{ end }}
       {{ end }}

Getting Errors as mentioned below:

Not sure what is wrong in the configmap. Can anyone help me with how to provision this? and help me spot what is missing or wrongly written in the configmap.

[cruxforum]

I was able to provision some parts of the above issue but facing issues as follows:

1. While provisioning alert rules, unable to provide the legendFormat it gives the following error:
   legendFormat: "{{persistentvolumeclaim}} : {{namespace}}"
   
   

also, unable to use humanize function in description while provisioning it using helm chart:

2. While provisioning contactpoints, unable to give custom template reference, getting below error:

title: |
  {{ template "crux-slack.title" . }}
text: |
  {{ template "crux-slack.text" . }}

3. Unable to provision templates, getting the below error:
   Unable to add {{- .Status | toUpper -}}

And Unable to add custom template due to the below error:

Can any one help with these issues?

[rloffelmacher]

Not sure if you are making use of Helm, but I ran across docs that specifically called out the double-curly braces as something which you need to escape in your alert definition. Maybe try that.