Telegram alerts: test message succeeded, but "400 Bad Request" on real alerts

I’m currently trying to set up telegram alerts. From notification channel edit page I can successfully send test message and read it from my telegram client. However, when I trying to test real alert from dashboard, I get log message

lvl=eror msg="Failed to send webhook" logger=alerting.notifier.telegram error="Webhook response status 400 Bad Request" webhook=my-telegram-bot

Checked with 2 different bots in different chats, same result - 400 error. Log level debug doesn’t help, no additional information on error in log files.

Grafana version=4.2.0 commit=349f3eb compiled=2017-03-22T15:06:41+0500, OS Ubuntu 16.04

Can anyone help with this issue?

Sounds like a network problem. Have you tried sending a message to Telegram from the machine where the Grafana server is installed?

Something like (change the botname and token):

curl -X POST https://api.telegram.org/bot368972338:BBF0s1EF2XqKPnAhc-4WLnEhCFU03xdXVlQ/sendMessage

Telegram API details: https://core.telegram.org/bots/api#making-requests

Curl from grafana server works well. See, test alert works well, only real alerts are failed, so I don’t think it’s network related issue.

I suppose it’s one of https://core.telegram.org/method/messages.sendMessage#return-errors errors, but actual response is not logged - so i can’t be sure.

It could be the message empty error. Is the notification message field in the triggered alert empty? This could be a bug in the telegram notifier.

No, I’ve double checked such cases ) Also, notification field is only part of the message: it automatically filled with metric data and alert details. So, as i see, even if I left notifications blank, message can not be empty.

I have russian locale on server, but anyway - test alerts are ok, and real alerts messages i’ve tested without cyrillic symbols, to avoid encoding errors.

Think i’ll try to catch alert packets with tcpdump, if it can handle https…

Think I’ve found issue. It’s < symbol in my notification: rps < 50. I quess, notification content is not escaped when message html is created. I replaced < with less than and it’s worked like a charm!

btw, thank’s for helping!

1 Like

Oh, sounds like a nasty bug. Nice catch. And thanks for creating the issue.

Hi,

I am having the same exact problem <Telegram test message succeeded, but “400 Bad Request” on real alerts> on Grafana v5.0.0-beta1 (commit: 3e2da1f) Ubuntu 16.04.3 LTS. No issue with parallel email notification. Same issue with/without text in notification text box.

Pls help. Thanks

t=2018-02-20T23:31:32+0800 lvl=info msg=“Sending telegram image notification” logger=alerting.notifier.telegram photo=/var/lib/grafana/png/DybUicP2w1NXWglxLZaU.png chat_id=-123456 bot_token=xxxxxxxxxxxxxxx
t=2018-02-20T23:31:34+0800 lvl=eror msg=“Failed to send webhook” logger=alerting.notifier.telegram error=“Webhook response status 400 Bad Request” webhook=TelegramBot

curl -X POST https://api.telegram.org/botxxxxxxxx/sendMessage?chat_id=-12345&text=Test
[1] 40827
{“ok”:false,“error_code”:400,“description”:“Bad Request: message text is empty”}

curl -X POST https://api.telegram.org/botxxxxxxxx/sendMessage
{“ok”:false,“error_code”:400,“description”:“Bad Request: message text is empty”}

Upon more troubleshooting, seems like message cannot be sent if the check box for sending image is checked.

Prior to this, I also uploaded my website SSL Cert to Telegram for the webhook, not sure if this is necessary.

This is probably this issue:

It was fixed in beta2.

Thanks.

Probably not, i am able to receive image via parallel email notifications. Also, i do not have legend on the right.

It is the same error as previously reported. Is there any text in the alert notification field?

image

I have text in the alert notification field. No symbols.

I have isolated the problem to ‘Include Image’ checkbox. I am able to receive the alerts and image in email notification no issue. So i guess the problem is with Telegram webhook with image.

Not really sure. When I searched for this, it seems to be a general error that Telegram returns that can mean a lot of different things.

Some related issues:

Can you try upgrading to the latest beta and see if that fix that is discussed in the first issue, fixes your problem.

@mcmxii a question, are you using inline images or an external link to your image (do you have an image store set up in your Grafana config or not).

@daniellee Thanks! Just realised that only email notification supports embedded image, while all other notifications e.g. Telegram require the image to be uploaded to external image store. That explains my case.

Sorry, I am not sure what is exactly inline image.

Is it necessary for the following command for Telegram notification to work?

curl -F “url=https://domain/webhooks/telegram” -F “certificate=@/somedirectory/cert.pem” https://api.telegram.org/bot$id/setWebhook

Thanks

Telegram should support inline (or embedded) images now:

and a followup issue:

I don’t know too much about the Telegram code but a quick glance shows that the api calls are either:

  • embedded/inline image: https://api.telegram.org/bot$id/sendPhoto
  • just a message: https://api.telegram.org/bot$id/sendMessage

It is a HTTP POST with a body that has the following fields if embedded/inline image:

  • chat_id
  • caption
  • photo

Hi, I’ve looked at the telegram notifier and the problem is that the sendPhoto endpoint for the inline message, does not have a “message” field, but a “caption” one instead, and this caption field is limited to 200 chars, but the length of the caption is not being checked at any moment here

A quick test hardcoding the caption sent to a couple words seems to be working right

Also, the test message is working because its using the text message format, but includes an image as an external link not as an embedded one.

Seperately, for text only alert sent to Telegram, when i click on the url link to Granana within Telegram app, it will show ‘unathorised’. No response when i click Sign In.

Perhaps we can have another checkbox in the Notification Channel settings to include or exclude URL link. Most of the time we only need the Predefined Text Message alert. This will reduce clutter in the Telegram chat.