Working configuration example for alerts templating - Telegram and Slack

Grafana Alerting
, , ,
15

Hello @bLd759, @usman.ahmad

An import UPDATE on my issue:
As I researched it a little bit deeper I found out that my particular problem was not only the Parse Mode. But changing anything on the edit page of that particular contact point fixed the error, but just for some time or partly (so some alerts literally squeezed through the limits some not). So, sending a test notification, changing the parse mode or a template - any action was making the contact point “alive” again.

After some days of searching I came across this issue, where a user faced with a similar problem: Webhook response status 400 Bad Request and provided a detailed and structured review with logs and reflection.
The point is that too many alerts generate a too large text message, which Telegram’s API can’t handle. The limit is 4096 symbols.

I checked that on my alerts removing a .ValueString part from the alert template, leaving just .Labels.alertname and that template unloading solved the problem permanently.

It turns out, that changing a Parse Mode may help, but first of all it is necessary to check if your Grafana is not sending too many alerts to Telegram at once (20+ in my case) so it doesn’t exceed the limit.

This is my template now for the contact points where a bunch of alerts is to be sent to the Telegram:

{{ define "tgbodydd" -}}
  {{ range . -}} 
    {{ with .Labels.alertname }}<b>{{ . }}</b>{{ end }}
  {{ end }}
{{ end }}

{{ define "tgdd" -}}
  {{ with .Alerts.Firing }}⚠️{{ template "tgbodydd" . }}{{ end }}
  {{ with .Alerts.Resolved }}✅OK{{ template "tgbodydd" . }}{{ end }}
{{ end }}

Unfortunately, the alerts using that template won’t be so informative as they were, but I am sure that won’t miss anything.

2 Likes