Grafana 9 Alert Migrations: legacy alert screenshot settings aren't automatically migrated to new alerts

Hi :grafana: users!

Grafana 9 adds support for images to grafana alerts! This is a feature that legacy alerting had - but it’s new for unified grafana alerting. That said, we want to point out that if you’re migrating from legacy alerting - there are a few configuration steps to add images in new alerts. These settings aren’t migrated from the legacy alerting section of your configuration file.

A short list of helpful docs to review for managing and configuring migrated grafana alerts:

Configuring images in new grafana alerts:

First, enable image rendering if it isn’t already!

If Grafana has been set up to use image rendering images in notifications can be turned on via the capture option in [unified_alerting.screenshots]:

# Enable screenshots in notifications. This option requires a remote HTTP image rendering service. Please
# see [rendering] for further configuration options.
capture = true

It is recommended that max_concurrent_screenshots is set to a value that is less than or equal to concurrent_render_request_limit. The default value for both max_concurrent_screenshots and concurrent_render_request_limit is 5:

# The maximum number of screenshots that can be taken at the same time. This option is different from
# concurrent_render_request_limit as max_concurrent_screenshots sets the number of concurrent screenshots
# that can be taken at the same time for all firing alerts where as concurrent_render_request_limit sets
# the total number of concurrent screenshots across all Grafana services.
max_concurrent_screenshots = 5

If Grafana has been set up to use an external image store, upload_external_image_storage should be set to true:

# Uploads screenshots to the local Grafana server or remote storage such as Azure, S3 and GCS. Please
# see [external_image_storage] for further configuration options. If this option is false, screenshots
# will be persisted to disk for up to temp_data_lifetime.
upload_external_image_storage = false

Restart Grafana for the changes to take effect.

Happy Alerting! :bell:

3 Likes

Nice to hear (read) that it is possible.

Unfortunetly I still can’t do it. I have Grafana and the renderer launched in a docker-compose.yml file and the option “[unified_alerting.screenshots]” does not appear in the grafana.ini file, so I cannot change it from enviornment variables of the docker-compose file. I have already set to true the variable “unified_alertin_enabled” and clustered mode for “…image_renderer_rendering_mode”. I’m missing something?

By the way, once running, should appear any option to select for doing the snapshot in the alert?

Thanks in advance!

@robone how are you building your docker image and what is the environment variable you’re trying to use?

@robone tried this out and was able to enable the flag with:

docker run -d --name=grafana9 -p 3000:3000 -e "GF_UNIFIED_ALERTING_SCREENSHOTS_CAPTURE=true" grafana/grafana

2 Likes

Hi @melori.arellano . Sorry for the delay, I had holiday. I’ve tried your instruction (in a new container) but didn’t work either, basically I cannot find that “send capture” option. Should the option be at the “contact point” window?. Here is my docker-compose conf:

#Renderer & alerts
GF_RENDERING_SERVER_URL: http://my_ip:8081/render
GF_RENDERING_CALLBACK_URL: http://my_ip:3000/
GF_LOG_FILTERS: rendering:debug
GF_UNIFIED_ALERTING_ENABLED: “true”
GF_ALERTING_ENABLED: “false”
GF_UNIFIED_ALERTING_SCREENSHOTS_CAPTURE: true
GF_FORCE_MIGRATION: “true”

1 Like

Should appear any option to select for doing the snapshot in the alert?

@omarmx90 check this doc page: https://grafana.com/docs/grafana/latest/alerting/migrating-alerts/migrating-legacy-alerts/

It states that: Unlike legacy dashboard alerts where images in notifications are enabled per contact point, images in notifications for Grafana Alerting must be enabled in the Grafana configuration, either in the configuration file or environment variables, and are enabled for either all or no contact points. Please refer to the documentation for images in notifications.

For the moment, I’m still not able to send those screenshots.

@robone & @omarmx90 I was testing this and noticed this detail:

Grafana does not support images for alert rules that are not associated with a panel. An alert rule is associated with a panel when it has both Dashboard UID and Panel ID annotations.

For example, this alert is associated with a panel:

You’ll also want to make sure that the alert is managed by your local grafana alertmanager (not external alertmanager, loki, or mimir).

1 Like

Well my objective is send images trough slack, but this still is not working, is there any example of implementation? Here is the ENV variables that I am using :

GF_FEATURE_TOGGLES_ENABLE: dashboardPreviews serviceAccounts savedItems panelTitleSearch
GF_EXTERNAL_IMAGE_STORAGE_PROVIDER: s3
GF_EXTERNAL_IMAGE_STORAGE_S3_REGION: region-xxxx
GF_EXTERNAL_IMAGE_STORAGE_S3_BUCKET: bucket-storage-s3
GF_EXTERNAL_IMAGE_STORAGE_S3_ACCESS_KEY: xxxxxxxx
GF_EXTERNAL_IMAGE_STORAGE_S3_SECRET_KEY: xxxxxxxx
GF_UNIFIED_ALERTING_SCREENSHOTS_CAPTURE: true
GF_UNIFIED_ALERTING_SCREENSHOTS_UPLOAD_EXTERNAL_IMAGE_STORAGE: true
GF_FORCE_MIGRATION: true
GF_RENDERING_SERVER_URL: http://my_ip:8081/render
GF_RENDERING_CALLBACK_URL: http://my_ip:3000/

@omarmx90 the screenshots will be sent automatically to the supported contact points once they’re configured.

Double check that the port you are using for the renderer it is the one matching in your variables. It was an error of mine. (Both services deployed in Docker).

1 Like

Hi @melori.arellano, the option [unified_alerting.screenshots] does not appear in the grafana.ini my server.
Thanks in advance!

@ndt1092 this option was added in grafana 9. If your grafana.ini file doesn’t already include it, you can add the settings. I’d recommend making a copy of your existing config file before making any changes.

Many thanks, I have more questions, about the alert, can “value” minimize it?, Alert received with too much content in “value”.
Value: [ var=‘B0’ metric=‘systemStateMemoryDeviceStatusCombined{hostname=“IdracDell-HN”, instance=“ip address”, job=“Idrac_DELL”, systemStatechassisIndex=“1”}’ labels={name=systemStateMemoryDeviceStatusCombined, hostname=IdracDell-HN, instance=ip address, job=Idrac_DELL, systemStatechassisIndex=1} value=4 ]

@ndt1092 I replied with some detailed instructions for formatting the alert values in this post:

hello @melori.arellano , thanks for your very helpful post. I’m trying to configure the image render on ECS, so I create the task definitions from Grafana and the Image render containers in the same service, defining almost the same env vars from @omarmx90 , but not defining any storage (since I want to sent it on Slack and don’t need to keep those images). Is there any other magical tip to make it work here?

@melori.arellano hi!
5 links in the first post of the topic is 404 error :frowning:

Oh no! Thanks for letting me know. I’ll get those updated!

Edit: They’re all updated now!

1 Like