I recently opted into the new unified alerting because I found the possibility of having multidimensional alerts very appealing and useful. However, I am currently struggling with setting up notifications (via Telegram).
What I have:
A system tracking temperatures.
What is working so far:
- I set up an alert based on the average temperature measured
- A notification channel that can send notifications via Telegram
- Notifications are successfully send when the temperature exceeds the defined threshold
What is not working
- No notification is sent when no data is available for the evaluation of the condition. This is both critical and was possible using the “legacy” alerting.
What am I using:
- Grafana version: 8.3.0
- operating system: Ubuntu 20.04
This problem was also discussed here: "NoData" alert not working in unified alerting · Issue #42774 · grafana/grafana · GitHub
I think the problem could be connected to the message content itself. At least for me (I am using Telegram), grafana seems to attempt to send a message, but ultimately fails to do so (see my logs):
t=2021-12-06T12:36:31+0000 lvl=info msg="sending telegram notification" logger=alerting.notifier.telegram chat_id=991107053
t=2021-12-06T12:36:31+0000 lvl=eror msg="Failed to send webhook" logger=alerting.notifier.telegram error="Webhook response status 400 Bad Request" webhook="Test Telegram Alert"
t=2021-12-06T12:36:31+0000 lvl=info msg="level=error component=dispatcher msg=\"Notify for alerts failed\" num_alerts=1 err=\"Test Telegram Alert/telegram[0]: notify retry canceled due to unrecoverable error after 1 attempts: Webhook response status 400 Bad Request\"" logger=alertmanager org=1
The problem can be “fixed” by setting a custom telegram message:
However, now only this message without any information about the alert itself is displayed.
