Embedding Panel Image in Email Alert Notification

Grafana Configuration
,
1

I am trying to have panel images embedded in my alert emails. I am running Grafana and Grafana Image Renderer as two seperate docker containers. Grafana completes its version check with Image Renderer and when I select a dashboard panel and execute a Direct link render image it executes correctly so calls to Image Renderer are executing properly. However my email alerts are not getting any images embedded when fired off and I am not seeing any rendering requests being executed by Grafana. What am I missing?

Grafana version: 10.2.3
Grafana Image Renderer version: 3.9.0

GF_AUTH_ANONYMOUS_ENABLED = true
GF_EXTERNAL_IMAGE_STORAGE_PROVIDER = local
GF_LOG_FILTERS = rendering:debug
GF_RENDERER_TOKEN =  XXXXX
GF_RENDERING_CALLBACK_URL = Grafana Host URL:3000
GF_RENDERING_SERVER_URL	= Image Renderer URL:8081/render
GF_SERVER_ROOT_URL = Grafana Host URL:3000
GF_UNIFIED_ALERTING.SCREENSHOTS_CAPTURE = true
GF_UNIFIED_ALERTING_ENABLED = true
GF_ALERTING_ENABLED = false
2

One obvious config issue: GF_UNIFIED_ALERTING.SCREENSHOTS_CAPTURE should be GF_UNIFIED_ALERTING_SCREENSHOTS_CAPTURE. Of course there can be more issues, not just this one.

3

Corrected that variable and now I am seeing the email render requests hit the Image Renderer but it is erroring out with “context deadline exceeded”.

Grafana Log

Blockquote INFO [01-09|13:38:10] Sending alerts to local notifier logger=ngalert.sender.router rule_uid=a75ec9fe-cbc8-45f1-86b4-741014e0f658 org_id=1 count=1
INFO [01-09|13:38:11] Request Completed logger=context userId=0 orgId=1 uname= method=GET path=/api/live/ws status=-1 remote_addr=172.17.0.1 time_ms=1836 duration=1.836664677s size=0 referer= handler=/api/live/ws
INFO [01-09|13:38:11] Initialized channel handler logger=live channel=grafana/dashboard/uid/_HA_0v87k address=grafana/dashboard/uid/_HA_0v87k
INFO [01-09|13:39:00] Rendering logger=rendering renderer=http path=“d-solo/_HA_0v87k/ring-availability?orgId=1&panelId=2”
DEBUG[01-09|13:39:00] calling remote rendering service logger=rendering renderer=http url=“http://192.168.1.112:8081/render?deviceScaleFactor=1.000000&domain=192.168.1.112&encoding=&height=500&renderKey=VYJqYXUf949JDr9cvzgNSiSoqetaz1No&timeout=10&timezone=&url=http%3A%2F%2F192.168.1.112%3A3000%2Fd-solo%2F_HA_0v87k%2Fring-availability%3ForgId%3D1%26panelId%3D2%26render%3D1&width=1000”
ERROR[01-09|13:39:10] Failed to send request to remote rendering service logger=rendering renderer=http error=“Get "http://192.168.1.112:8081/render?deviceScaleFactor=1.000000&domain=192.168.1.112&encoding=&height=500&renderKey=VYJqYXUf949JDr9cvzgNSiSoqetaz1No&timeout=10&timezone=&url=http%3A%2F%2F192.168.1.112%3A3000%2Fd-solo%2F_HA_0v87k%2Fring-availability%3ForgId%3D1%26panelId%3D2%26render%3D1&width=1000\”: context deadline exceeded"
ERROR[01-09|13:39:10] Failed to delete render key logger=rendering error=“context deadline exceeded”
WARN [01-09|13:39:10] Failed to take an image logger=ngalert.state.manager rule_uid=a75ec9fe-cbc8-45f1-86b4-741014e0f658 org_id=1 instance= dashboard=_HA_0v87k panel=2 error="failed to take screenshot: [rendering.serverTimeout] "
INFO [01-09|13:39:11] Sending alerts to local notifier logger=ngalert.sender.router rule_uid=a75ec9fe-cbc8-45f1-86b4-741014e0f658 org_id=1 count=1
INFO [01-09|13:39:11] Request Completed logger=context userId=0 orgId=1 uname= method=GET path=/api/live/ws status=-1 remote_addr=172.17.0.1 time_ms=753 duration=753.845544ms size=0 referer= handler=/api/live/ws

Image Renderer

Blockquote 024-01-09T13:38:13.611581844Z ERR Request failed | stack=Error: Request aborted
at onaborted (/usr/src/app/node_modules/express/lib/response.js:1052:15)
at Immediate._onImmediate (/usr/src/app/node_modules/express/lib/response.js:1094:9)
at process.processImmediate (node:internal/timers:476:21) url=/render?deviceScaleFactor=1.000000&domain=192.168.1.112&encoding=&height=500&renderKey=cV6DnW1s7u1olUv25yxFAn8uO38dsRwD&timeout=10&timezone=&url=http%3A%2F%2F192.168.1.112%3A3000%2Fd-solo%2F_HA_0v87k%2Fring-availability%3ForgId%3D1%26panelId%3D2%26render%3D1&width=1000
2024-01-09T13:39:14.398467055Z ERR Request failed | stack=Error: Request aborted
at onaborted (/usr/src/app/node_modules/express/lib/response.js:1052:15)
at Immediate._onImmediate (/usr/src/app/node_modules/express/lib/response.js:1094:9)
at process.processImmediate (node:internal/timers:476:21) url=/render?deviceScaleFactor=1.000000&domain=192.168.1.112&encoding=&height=500&renderKey=VYJqYXUf949JDr9cvzgNSiSoqetaz1No&timeout=10&timezone=&url=http%3A%2F%2F192.168.1.112%3A3000%2Fd-solo%2F_HA_0v87k%2Fring-availability%3ForgId%3D1%26panelId%3D2%26render%3D1&width=1000

4

Okay, got it working! Added following lines to config:

GF_UNIFIED_ALERTING_EVALUATION_TIMEOUT = 90s
GF_UNIFIED_ALERTING_SCREENSHOTS_CAPTURE_TIMEOUT = 30s

Note: Max value allowed for SCREENSHOTS_CAPTURE_TIMEOUT is 30s.