I’m using Grafana OSS on Ubuntu 22.04, and I have installed the grafana image renderer plugin with
grafana-cli plugins install grafana-image-renderer
After that, I changed /etc/grafana/grafana.ini added these:
[undefined_alerting.screenshots]
# Enable screenshots in notifications. You must have either installed the Grafana image rendering
# plugin, or set up Grafana to use a remote rendering service.
# For more information on configuration options, refer to [rendering].
capture = false
# Uploads screenshots to the local Grafana server or remote storage such as Azure, S3 and GCS. Please
# see [external_image_storage] for further configuration options. If this option is false, screenshots
# will be persisted to disk for up to temp_data_lifetime.
upload_external_image_storage = false
[log]
level = debug
I can confirm that the image rendering plugin works, because I can open any panel, go to shared / link and there is a “Direct link rendered image” link. If I open that link, then I’m redirected to an URL that contains a single png image (screenshot) of the panel.
However, if an alert is generated, then the image is not attached to the email notification, and is is also missing from the telegram notification. Here is the debug log (I have replaced some names to hide my host name):
logger=ngalert.state.manager rule_uid=fd794b53-599f-4471-97a4-69775dc2a5f9 org_id=2 t=2023-08-19T17:00:09.177415595Z level=debug msg="State manager processing evaluation results" resultCount=2
logger=ngalert.state.manager rule_uid=fd794b53-599f-4471-97a4-69775dc2a5f9 org_id=2 instance="datasource_uid=__expr__, ref_id=above 10 minutes,max_total_elapsed" t=2023-08-19T17:00:09.177770144Z level=debug msg="Setting next state" handler=resultNoData
logger=ngalert.state.manager rule_uid=fd794b53-599f-4471-97a4-69775dc2a5f9 org_id=2 instance="datasource_uid=__expr__, ref_id=above 10 minutes,max_total_elapsed" t=2023-08-19T17:00:09.177840375Z level=debug msg="Execution no data state is Alerting" handler=resultAlerting previous_handler=resultNoData
logger=ngalert.state.manager rule_uid=fd794b53-599f-4471-97a4-69775dc2a5f9 org_id=2 instance="datasource_uid=__expr__, ref_id=above 10 minutes,max_total_elapsed" t=2023-08-19T17:00:09.177867986Z level=debug msg="Keeping state" state=Alerting previous_ends_at=2023-08-19T17:10:00Z next_ends_at=2023-08-19T17:15:00Z
logger=ngalert.image rule_uid=fd794b53-599f-4471-97a4-69775dc2a5f9 org_id=2 dashboard=bf7257eb-605a-4d2a-9ba0-cfbd346d6f10 panel=2 t=2023-08-19T17:00:09.177909213Z level=debug msg="Requesting screenshot"
logger=ngalert.state.manager rule_uid=fd794b53-599f-4471-97a4-69775dc2a5f9 org_id=2 instance="datasource_uid=ae62ee76-a746-4704-935f-309984735629, ref_id=total_elapsed" t=2023-08-19T17:00:09.178151232Z level=debug msg="Setting next state" handler=resultNoData
logger=ngalert.state.manager rule_uid=fd794b53-599f-4471-97a4-69775dc2a5f9 org_id=2 instance="datasource_uid=ae62ee76-a746-4704-935f-309984735629, ref_id=total_elapsed" t=2023-08-19T17:00:09.178254815Z level=debug msg="Execution no data state is Alerting" handler=resultAlerting previous_handler=resultNoData
logger=ngalert.state.manager rule_uid=fd794b53-599f-4471-97a4-69775dc2a5f9 org_id=2 instance="datasource_uid=ae62ee76-a746-4704-935f-309984735629, ref_id=total_elapsed" t=2023-08-19T17:00:09.17831121Z level=debug msg="Keeping state" state=Alerting previous_ends_at=2023-08-19T17:10:00Z next_ends_at=2023-08-19T17:15:00Z
logger=ngalert.image rule_uid=fd794b53-599f-4471-97a4-69775dc2a5f9 org_id=2 dashboard=bf7257eb-605a-4d2a-9ba0-cfbd346d6f10 panel=2 t=2023-08-19T17:00:09.178356985Z level=debug msg="Requesting screenshot"
logger=ngalert.state.manager rule_uid=fd794b53-599f-4471-97a4-69775dc2a5f9 org_id=2 t=2023-08-19T17:00:09.178419912Z level=debug msg="Saving alert states" count=2 max_state_save_concurrency=1
logger=ngalert.state.manager rule_uid=fd794b53-599f-4471-97a4-69775dc2a5f9 org_id=2 t=2023-08-19T17:00:09.196640161Z level=debug msg="Saving alert states done" count=2 max_state_save_concurrency=1 duration=18.219899ms
logger=ngalert.sender.router rule_uid=fd794b53-599f-4471-97a4-69775dc2a5f9 org_id=2 t=2023-08-19T17:00:09.196861613Z level=info msg="Sending alerts to local notifier" count=2
logger=alertmanager org=2 t=2023-08-19T17:00:09.196928416Z level=debug component=alertmanager orgID=2 msg="Putting alert" alert=someuser@anotherhost.com[bf50960][active] starts_at=2023-07-21T20:00:00Z ends_at=2023-08-19T17:15:00Z
logger=alertmanager org=2 t=2023-08-19T17:00:09.197031659Z level=debug component=alertmanager orgID=2 msg="Putting alert" alert=someuser@anotherhost.com[994e4cd][active] starts_at=2023-07-21T20:00:00Z ends_at=2023-08-19T17:15:00Z
logger=alertmanager org=2 t=2023-08-19T17:00:09.197169625Z level=debug component=alertmanager orgID=2 component=dispatcher msg="Received alert" alert=someuser@anotherhost.com[bf50960][active]
logger=alertmanager org=2 t=2023-08-19T17:00:09.19722615Z level=debug component=alertmanager orgID=2 component=dispatcher msg="Received alert" alert=someuser@anotherhost.com[994e4cd][active]
logger=ngalert.scheduler t=2023-08-19T17:00:10.00198056Z level=debug msg="No changes detected. Skip updating"
logger=ngalert.state.manager t=2023-08-19T17:00:19.55481183Z level=debug msg="Recording state cache metrics" now=2023-08-19T17:00:19.554723846Z
logger=ngalert.scheduler t=2023-08-19T17:00:20.001758825Z level=debug msg="No changes detected. Skip updating"
logger=ngalert.scheduler t=2023-08-19T17:00:30.002206989Z level=debug msg="No changes detected. Skip updating"
logger=secrets t=2023-08-19T17:00:34.532924057Z level=debug msg="Removing expired data keys from cache..."
logger=secrets t=2023-08-19T17:00:34.533113418Z level=debug msg="Removing expired data keys from cache finished successfully"
logger=ngalert.state.manager t=2023-08-19T17:00:34.55497766Z level=debug msg="Recording state cache metrics" now=2023-08-19T17:00:34.554957834Z
logger=ngalert.sender.router t=2023-08-19T17:00:34.564971933Z level=debug msg="Attempting to sync admin configs" count=0
logger=ngalert.sender.router t=2023-08-19T17:00:34.565099641Z level=debug msg="Finish of admin configuration sync"
logger=ngalert.multiorg.alertmanager t=2023-08-19T17:00:34.58213631Z level=debug msg="synchronizing Alertmanagers for orgs"
logger=alertmanager org=2 t=2023-08-19T17:00:34.582858593Z level=debug component=alertmanager orgID=1 component=dispatcher aggrGroup="{}/{rule_uid=\"cIztBqi4z\"}/{rule_uid=\"cIztBqi4z\"}:{alertname=\"SomeCompany-slave lag on not.telling.com (low lag) alert\", grafana_folder=\"General Alerting\"}" msg=flushing alerts="[SomeCompany-slave lag on not.telling.com (low lag) alert[2874d1c][resolved]]"
logger=alertmanager org=2 t=2023-08-19T17:00:34.582904328Z level=debug component=alertmanager orgID=1 component=dispatcher aggrGroup="{}/{rule_uid=\"cIztBqi4z\"}/{rule_uid=\"cIztBqi4z\"}:{alertname=\"SomeCompany-slave lag on not.telling.com (low lag) alert\", grafana_folder=\"General Alerting\"}" msg=flushing alerts="[SomeCompany-slave lag on not.telling.com (low lag) alert[2874d1c][resolved]]"
logger=alertmanager org=2 t=2023-08-19T17:00:34.584087937Z level=debug msg="neither config nor template have changed, skipping configuration sync."
logger=alertmanager org=2 t=2023-08-19T17:00:34.585339995Z level=debug msg="neither config nor template have changed, skipping configuration sync."
logger=ngalert.multiorg.alertmanager t=2023-08-19T17:00:34.585846936Z level=debug msg="done synchronizing Alertmanagers for orgs"
logger=notifications t=2023-08-19T17:00:34.58740378Z level=debug msg="Sending webhook" url=https://api.telegram.org/bot19999999:AAF-65_Uascasdfasfacs7uyU/sendMessage httpmethod=POST
logger=notifications t=2023-08-19T17:00:34.788857813Z level=debug msg="Webhook succeeded" url=https://api.telegram.org/bot19999999:AAF-65_Uascasdfasfacs7uyU/sendMessage statuscode="200 OK"
logger=alertmanager org=2 t=2023-08-19T17:00:34.789032597Z level=debug component=alertmanager orgID=1 component=dispatcher receiver=Grafana_My_Telegram integration=telegram[0] msg="Notify success" attempts=1
logger=alertmanager org=2 t=2023-08-19T17:00:34.860666308Z level=debug component=alertmanager orgID=1 component=dispatcher receiver="E-mail admins" integration=email[0] msg="Notify success" attempts=1
logger=alertmanager org=2 t=2023-08-19T17:00:38.213986887Z level=debug component=alertmanager orgID=1 component=dispatcher aggrGroup="{}/{rule_uid=\"xHzpfqmVk\"}/{rule_uid=\"xHzpfqmVk\"}:{alertname=\"AnotherCompany-slave lag on not.telling.com (low lag) alert\", grafana_folder=\"General Alerting\"}" msg=flushing alerts="[AnotherCompany-slave lag on not.telling.com (low lag) alert[815153b][resolved]]"
logger=alertmanager org=2 t=2023-08-19T17:00:38.214567986Z level=debug component=alertmanager orgID=1 component=dispatcher aggrGroup="{}/{rule_uid=\"xHzpfqmVk\"}/{rule_uid=\"xHzpfqmVk\"}:{alertname=\"AnotherCompany-slave lag on not.telling.com (low lag) alert\", grafana_folder=\"General Alerting\"}" msg=flushing alerts="[AnotherCompany-slave lag on not.telling.com (low lag) alert[815153b][resolved]]"
logger=notifications t=2023-08-19T17:00:38.215002023Z level=debug msg="Sending webhook" url=https://api.telegram.org/bot19999999:AAF-65_Uascasdfasfacs7uyU/sendMessage httpmethod=POST
logger=notifications t=2023-08-19T17:00:38.334841607Z level=debug msg="Webhook succeeded" url=https://api.telegram.org/bot19999999:AAF-65_Uascasdfasfacs7uyU/sendMessage statuscode="200 OK"
logger=alertmanager org=2 t=2023-08-19T17:00:38.335080371Z level=debug component=alertmanager orgID=1 component=dispatcher receiver=Grafana_My_Telegram integration=telegram[0] msg="Notify success" attempts=1
logger=alertmanager org=2 t=2023-08-19T17:00:38.388631789Z level=debug component=alertmanager orgID=1 component=dispatcher receiver="E-mail admins" integration=email[0] msg="Notify success" attempts=1
logger=ngalert.scheduler t=2023-08-19T17:00:40.001795018Z level=debug msg="No changes detected. Skip updating"
I can see that the the screenshot was requested:
logger=ngalert.image rule_uid=fd794b53-599f-4471-97a4-69775dc2a5f9 org_id=2 dashboard=bf7257eb-605a-4d2a-9ba0-cfbd346d6f10 panel=2 t=2023-08-19T17:00:09.177909213Z level=debug msg="Requesting screenshot"
There is no error message in the log. There is nothing else in the log - although I turned on level=debug, and log filters are empty. And yet I’m not getting any images.
I went through the troubleshooting steps here: Use images in notifications | Grafana documentation but all of the steps got a big green checkmark.
What is missing please?