One of the biggest issues with monitoring and alerting systems is that alerting just goes bad and people end up ignoring it. That can be because of false positives, but also because alert messages are unclear.
Upgrading from Grafana 8.4.4 to 9.* automatically opted me in to unified alerts. I was going to move to that anyway and all the alerts technically work… but the default alert message is a mess. For example, here is a Slack alert from the old Grafana; please excuse the heavy obfuscation and tyops:
Here’s the same alert from Grafana 9.2.0:
This is the kind of message people absolutely ignore.
I’ve gone through the templating system and, honestly, it’s a huge mess, especially since I’m not familiar with Go, and being a template means it’ll not really be easily human-readable (i.e., no indenting, etc).
I did manage to struggle through and finally got this:
…which apparently I can’t attach because I’m a new user. I’ve added it in a comment below.
It’s pretty close but the problem is that I can’t use any Slack markdown, can’t use a title (because it repeats multiple times), and if I have multiple hosts, I can’t show those because the hosts are in the ValueString variable which is not really processable easily. I’ve seen many, many posts about this variable and no actual solutions. In fact there are some conversations ending with “Oh well”.
So my questions are, basically, can I just tell it to use the same “template” it did before unified alerting?
Barring that, is there a better way to process the string ValueString as an array? None of the functions I tried on it work, and I had to do some dirty, dirty regex to compress it down to one single hostname.
Here is the template I’m using now:
[{{.Status | toUpper }}] {{ .Labels.alertname }}
{{ if gt (len .ValueString) 0 }} {{ $hostname := reReplaceAll "^(.*?)hostname=" "" .ValueString }} {{ $hostname := reReplaceAll ",.*$" "" $hostname }} {{ if gt (len $hostname) 0 }}
Hostname: {{ $hostname }}{{ end }}{{ end }}
{{ if gt (len .Annotations) 0 }}
{{ .Annotations.message }}{{ end }}
{{ end }}
FIRING:{{ " " }}
RESOLVED:{{ " " }}