Telegram alerts: test message succeeded, but "400 Bad Request" on real alerts

evagrafana · May 5, 2017, 5:30am

I’m currently trying to set up telegram alerts. From notification channel edit page I can successfully send test message and read it from my telegram client. However, when I trying to test real alert from dashboard, I get log message

lvl=eror msg="Failed to send webhook" logger=alerting.notifier.telegram error="Webhook response status 400 Bad Request" webhook=my-telegram-bot

Checked with 2 different bots in different chats, same result - 400 error. Log level debug doesn’t help, no additional information on error in log files.

Grafana version=4.2.0 commit=349f3eb compiled=2017-03-22T15:06:41+0500, OS Ubuntu 16.04

Can anyone help with this issue?

daniellee · May 5, 2017, 10:03am

Sounds like a network problem. Have you tried sending a message to Telegram from the machine where the Grafana server is installed?

Something like (change the botname and token):

curl -X POST https://api.telegram.org/bot368972338:BBF0s1EF2XqKPnAhc-4WLnEhCFU03xdXVlQ/sendMessage

Telegram API details: https://core.telegram.org/bots/api#making-requests

evagrafana · May 5, 2017, 10:48am

Curl from grafana server works well. See, test alert works well, only real alerts are failed, so I don’t think it’s network related issue.

I suppose it’s one of https://core.telegram.org/method/messages.sendMessage#return-errors errors, but actual response is not logged - so i can’t be sure.

daniellee · May 5, 2017, 10:54am

It could be the message empty error. Is the notification message field in the triggered alert empty? This could be a bug in the telegram notifier.

evagrafana · May 5, 2017, 11:58am

No, I’ve double checked such cases ) Also, notification field is only part of the message: it automatically filled with metric data and alert details. So, as i see, even if I left notifications blank, message can not be empty.

I have russian locale on server, but anyway - test alerts are ok, and real alerts messages i’ve tested without cyrillic symbols, to avoid encoding errors.

Think i’ll try to catch alert packets with tcpdump, if it can handle https…

evagrafana · May 5, 2017, 12:26pm

Think I’ve found issue. It’s < symbol in my notification: rps < 50. I quess, notification content is not escaped when message html is created. I replaced < with less than and it’s worked like a charm!

btw, thank’s for helping!

daniellee · May 5, 2017, 1:09pm

Oh, sounds like a nasty bug. Nice catch. And thanks for creating the issue.

mcmxii · February 20, 2018, 4:07pm

Hi,

I am having the same exact problem <Telegram test message succeeded, but “400 Bad Request” on real alerts> on Grafana v5.0.0-beta1 (commit: 3e2da1f) Ubuntu 16.04.3 LTS. No issue with parallel email notification. Same issue with/without text in notification text box.

Pls help. Thanks

t=2018-02-20T23:31:32+0800 lvl=info msg=“Sending telegram image notification” logger=alerting.notifier.telegram photo=/var/lib/grafana/png/DybUicP2w1NXWglxLZaU.png chat_id=-123456 bot_token=xxxxxxxxxxxxxxx
t=2018-02-20T23:31:34+0800 lvl=eror msg=“Failed to send webhook” logger=alerting.notifier.telegram error=“Webhook response status 400 Bad Request” webhook=TelegramBot

curl -X POST https://api.telegram.org/botxxxxxxxx/sendMessage?chat_id=-12345&text=Test
[1] 40827
{“ok”:false,“error_code”:400,“description”:“Bad Request: message text is empty”}

curl -X POST https://api.telegram.org/botxxxxxxxx/sendMessage
{“ok”:false,“error_code”:400,“description”:“Bad Request: message text is empty”}

mcmxii · February 21, 2018, 6:03am

Upon more troubleshooting, seems like message cannot be sent if the check box for sending image is checked.

Prior to this, I also uploaded my website SSL Cert to Telegram for the webhook, not sure if this is necessary.

daniellee · February 21, 2018, 8:28am

This is probably this issue:

github.com/grafana/grafana

Image render timeout for graph panel with legend shown as table on the right and with at least one value

opened 01:32PM - 16 Jan 18 UTC

closed 11:42AM - 15 Feb 18 UTC

nklebedev

type/bug

Grafana version is 5.0.0-11204pre1. Direct image rendering (via phanotomjs) i…s resulted in timeout for any panel if this panel has legend on the right, formatted as table and has at least one value (min, max, avg, current or total). Therefore, images for such kind of panels can't be embedded into alerts. From log: Loading a web page: http://localhost:3000/dashboard-solo/db/<mydashboard>... Timeout error. You can set timeout in seconds with &timeout url parameter I made some investigation on the issue by adding some logging to the *phantomjs/render.js*. As far as I can see the **rootScope.panelsRendered** is always undefined in line 61 and rendering finally interrupted by timeout.

It was fixed in beta2.

mcmxii · February 21, 2018, 8:40am

Thanks.

Probably not, i am able to receive image via parallel email notifications. Also, i do not have legend on the right.

daniellee · February 21, 2018, 8:48am

It is the same error as previously reported. Is there any text in the alert notification field?

mcmxii · February 21, 2018, 9:20am

I have text in the alert notification field. No symbols.

I have isolated the problem to ‘Include Image’ checkbox. I am able to receive the alerts and image in email notification no issue. So i guess the problem is with Telegram webhook with image.

daniellee · February 21, 2018, 10:11am

Not really sure. When I searched for this, it seems to be a general error that Telegram returns that can mean a lot of different things.

Some related issues:

Can you try upgrading to the latest beta and see if that fix that is discussed in the first issue, fixes your problem.

daniellee · February 21, 2018, 10:22am

@mcmxii a question, are you using inline images or an external link to your image (do you have an image store set up in your Grafana config or not).

mcmxii · February 21, 2018, 1:37pm

@daniellee Thanks! Just realised that only email notification supports embedded image, while all other notifications e.g. Telegram require the image to be uploaded to external image store. That explains my case.

Sorry, I am not sure what is exactly inline image.

mcmxii · February 21, 2018, 1:39pm

Is it necessary for the following command for Telegram notification to work?

curl -F “url=https://domain/webhooks/telegram” -F “certificate=@/somedirectory/cert.pem” https://api.telegram.org/bot$id/setWebhook

Thanks

daniellee · February 21, 2018, 2:03pm

Telegram should support inline (or embedded) images now:

github.com/grafana/grafana

Add support for inline image uploads to telegram notifier plugin

grafana:master ← rburchell:master

opened 03:43PM - 22 Nov 17 UTC

rburchell

+89 -24

As requested in #8246, and initially started in #8373 by @wf1nder, add support f…or inline image uploads to the telegram notifier plugin. This implementation is different: * rather than directly using net/http, it still uses SendWebhookSync in an attempt to reuse more of core functionality * as a result, rather than using the existing simplejson payload, it uses multipart for both text and image uploads * there are (effectively) no changes to message content included in this pull request, with the exception that image messages do not have HTML tags as they do not appear to be supported. Any changes to message content should be done as a separate pull request. There is still some slight duplication in how the payloads are created, but in my opinion, it likely isn't worth the effort to share any of this code: image and text messages have different API with different parameters, and it isn't a large amount of code, after all. No unit tests have been added at this time, as it is not clear to me precisely what should be tested, and how, especially as this would require connection to a live external service.

and a followup issue:

https://github.com/grafana/grafana/issues/10631

I don’t know too much about the Telegram code but a quick glance shows that the api calls are either:

embedded/inline image: https://api.telegram.org/bot$id/sendPhoto
just a message: https://api.telegram.org/bot$id/sendMessage

It is a HTTP POST with a body that has the following fields if embedded/inline image:

chat_id
caption
photo

pho · February 21, 2018, 7:46pm

Hi, I’ve looked at the telegram notifier and the problem is that the sendPhoto endpoint for the inline message, does not have a “message” field, but a “caption” one instead, and this caption field is limited to 200 chars, but the length of the caption is not being checked at any moment here

A quick test hardcoding the caption sent to a couple words seems to be working right

Also, the test message is working because its using the text message format, but includes an image as an external link not as an embedded one.

mcmxii · February 24, 2018, 12:52am

Seperately, for text only alert sent to Telegram, when i click on the url link to Granana within Telegram app, it will show ‘unathorised’. No response when i click Sign In.

Perhaps we can have another checkbox in the Notification Channel settings to include or exclude URL link. Most of the time we only need the Predefined Text Message alert. This will reduce clutter in the Telegram chat.