Webhook issues after upgrade from 8.x to 9.4.3 - bad silence url encoding breaks Symphony parsing

jasonpcom · March 20, 2023, 4:43pm

After upgrading to Grafana 9.4.3 from 8.3.4

The webhook message that is generated has a badly encoded silence URL which causes external system being called to fail to parse.

message contains

Silence:  https://f-uat-2.systems.uk.somewhere:1234/alerting/silence/new?alertmanager=grafana&matcher=alertname%3DH+Instances+below+expected+count+for+2m+period+Alert&matcher=grafana_folder%3DD+System&matcher=rule_uid%3D5HTjtV-4k

Causes

 {"code":400,"message":"Failed to parse MessageML: Invalid messageML: The reference to entity \"matcher\" must end with the ';' delimiter."}

The silence URL is badly encoded, can the generation of Silence URL sent in webhook be suppressed? or correct encoding so uses &amp rather than &

jasonpcom · March 21, 2023, 12:43pm

It looks like the default alert template in 9.4.3

github.com

grafana/alerting/blob/main/templates/default_template.go

package templates

import (
	"os"
	"testing"

	"github.com/prometheus/alertmanager/template"
	"github.com/stretchr/testify/require"
)

const (
	DefaultMessageTitleEmbed = `{{ template "default.title" . }}`
	DefaultMessageEmbed      = `{{ template "default.message" . }}`
)

var DefaultTemplateString = `
{{ define "__subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ if gt (.Alerts.Resolved | len) 0 }}, RESOLVED:{{ .Alerts.Resolved | len }}{{ end }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}{{ end }}

{{ define "__text_values_list" }}{{ if len .Values }}{{ $first := true }}{{ range $refID, $value := .Values -}}
{{ if $first }}{{ $first = false }}{{ else }}, {{ end }}{{ $refID }}={{ $value }}{{ end -}}

This file has been truncated. show original

After alert expanded to give message causes the Symphony MessageML parser ( a secure encrypted chat-based collaboration product ) to fail

mjacobson · March 23, 2023, 6:03am

Hi @jasonpcom! I’m not entirely sure what you mean by the URL being badly encoded, it looks correctly encoded to me. That being said, as you’ve seen, the silence is part of the default message for an alert but this can be be overriden by defining your own custom template and then using that in the webhook contact point.

You can find more detailed documentation and examples here: Customize notifications | Grafana documentation

jasonpcom · March 23, 2023, 3:36pm

Your correct - the problem is that in upgrading to 9.x the default alert message template has changed which now includes an url link to ‘silence’ the alert. The format of the url causes the Symphony ( third party secure chat product used by finance community ) webhook parser to barf.

So agree not an issue with url encoding as such. Fix is to supply our own template that removes the ‘silence’ alert link until Symphony fix their webhook message parser.