Webhook issues after upgrade from 8.x to 9.4.3 - bad silence url encoding breaks Symphony parsing

After upgrading to Grafana 9.4.3 from 8.3.4

The webhook message that is generated has a badly encoded silence URL which causes external system being called to fail to parse.

message contains

Silence:  https://f-uat-2.systems.uk.somewhere:1234/alerting/silence/new?alertmanager=grafana&matcher=alertname%3DH+Instances+below+expected+count+for+2m+period+Alert&matcher=grafana_folder%3DD+System&matcher=rule_uid%3D5HTjtV-4k


 {"code":400,"message":"Failed to parse MessageML: Invalid messageML: The reference to entity \"matcher\" must end with the ';' delimiter."}

The silence URL is badly encoded, can the generation of Silence URL sent in webhook be suppressed? or correct encoding so uses &amp rather than &

It looks like the default alert template in 9.4.3

After alert expanded to give message causes the Symphony MessageML parser ( a secure encrypted chat-based collaboration product ) to fail

Hi @jasonpcom! I’m not entirely sure what you mean by the URL being badly encoded, it looks correctly encoded to me. That being said, as you’ve seen, the silence is part of the default message for an alert but this can be be overriden by defining your own custom template and then using that in the webhook contact point.

You can find more detailed documentation and examples here: Customize notifications | Grafana documentation

1 Like

Your correct - the problem is that in upgrading to 9.x the default alert message template has changed which now includes an url link to ‘silence’ the alert. The format of the url causes the Symphony ( third party secure chat product used by finance community ) webhook parser to barf.

So agree not an issue with url encoding as such. Fix is to supply our own template that removes the ‘silence’ alert link until Symphony fix their webhook message parser.