Grafana Image Renderer - Partially working?

luke · September 1, 2023, 11:21am

Hi, I’m in the process of configuring the Grafana Image Renderer but I’m getting some unusual behaviour.

Grafana is deployed via helm chart and so imageRender is configured as such below

imageRenderer:
  enabled: true
  replicas: 1
  autoscaling:
    enabled: false
  image:
    repository: docker.io/grafana/grafana-image-renderer
    tag: 3.1.0
    pullPolicy: Always
  env:
    HTTP_HOST: "0.0.0.0"
    AUTH_TOKEN: "testy123"
    RENDERING_VERBOSE_LOGGING: true
    RENDERING_DUMPIO: true
    RENDERING_ARGS: --no-sandbox,--disable-gpu
  service:
    enabled: true
    port: 80
    targetPort: 8081
  networkPolicy:
    limitIngress: false

Grafana configuration for this:

    [rendering]
    renderer_token = testy123
    [unified_alerting]
    enabled = true
    ha_peers = grafana:9094
    [unified_alerting.screenshots]
    capture = true
    upload_external_image_storage = false
    [external_image_storage]
    provider = local
    upload_external_image_storage = false

I’m currently unable to tell what the problem might be. At the moment I’m able to go to the panel > share and use “Direct link rendered image” and I can sometimes see log entries for panel rendering but not always when an alert is triggered or when I fire a test alert to my slack contact point (with dashboard and panel IDs), nothing seems to happen with rendering.

I’m unable to find any error logs at the moment and I’ve put Grafana into debug mode.

If anyone has any advice, would be greatly appreciated. The slack contact point is also using a bot account with below permissions.

incoming-webhook - Post messages to specific channels in Slack
files:write - Upload, edit and delete files as Grafana alert

I’m confused as I can render an image using the share option (log to prove):

logger=rendering renderer=http t=2023-09-01T11:20:35.283269072Z level=debug msg="calling remote rendering service" url="http://grafana-image-renderer.monitoring:80/...[REMOVED]

luke · September 2, 2023, 8:38am

Bumping up as I could do with some help here

luke · September 4, 2023, 8:00pm

And another bump, hoping someone here might have an idea?

luke · September 11, 2023, 8:15am

another bump

georgerobinson · September 11, 2023, 8:20am

Can you run Grafana with debug logging turned on? Troubleshooting | Grafana documentation

luke · September 11, 2023, 8:31am

Debug mode is on, nothing extra is logged to indicate the problem

georgerobinson · September 11, 2023, 9:21am

Can you share the logs for ngalert.image?

luke · September 11, 2023, 10:24am

tlogger=ngalert.image rule_uid=kvbbldC4k org_id=1 t=2023-09-11T09:21:03.395599115Z level=debug msg="Cannot take screenshot for alert rule as it is not associated with a dashboard"
logger=ngalert.image rule_uid=kvbbldC4k org_id=1 t=2023-09-11T09:20:08.396127525Z level=debug msg="Cannot take screenshot for alert rule as it is not associated with a dashboard"
logger=ngalert.image rule_uid=kvbbldC4k org_id=1 t=2023-09-11T09:20:05.374930034Z level=debug msg="Cannot take screenshot for alert rule as it is not associated with a dashboard"
logger=ngalert.image rule_uid=flwElkiVk org_id=1 dashboard=yLU8ue6Mz panel=20 t=2023-09-11T08:47:07.552492709Z level=debug msg="Saved image" token=4d0739aa-a1cd-430d-9aca-5b1683f46685
logger=ngalert.image rule_uid=flwElkiVk org_id=1 dashboard=yLU8ue6Mz panel=20 t=2023-09-11T08:47:07.546822759Z level=debug msg="Took screenshot" path=/var/lib/grafana/png/jtFI1jBthq63OJF0RjUE.png
logger=ngalert.image rule_uid=flwElkiVk org_id=1 dashboard=yLU8ue6Mz panel=20 t=2023-09-11T08:47:06.711916892Z level=debug msg="Saved image" token=faa5cf6a-d6af-4858-8f21-f8f7725d9bd1
logger=ngalert.image rule_uid=flwElkiVk org_id=1 dashboard=yLU8ue6Mz panel=20 t=2023-09-11T08:47:06.70232212Z level=debug msg="Took screenshot" path=/var/lib/grafana/png/Jay5Vv5DR7IDAfIQQO40.png
logger=ngalert.image rule_uid=flwElkiVk org_id=1 dashboard=yLU8ue6Mz panel=20 t=2023-09-11T08:47:02.728371299Z level=debug msg="Requesting screenshot"
logger=ngalert.image rule_uid=flwElkiVk org_id=1 dashboard=yLU8ue6Mz panel=20 t=2023-09-11T08:47:00.504233751Z level=debug msg="Requesting screenshot"

The “cannot take screenshot for alert rule…” is fine as that’s for a dashboard with nothing set, otherwise everything looks fine.

I’ve just done a request which didn’t appear in ngalert.image, I had to search for “render” which then showed up

logger=rendering renderer=http t=2023-09-11T10:18:10.645807108Z level=debug msg="calling remote rendering service" url=http://grafana-image-renderer.monitoring:80/render/version

The other thing I’m wondering is this:

DevTools listening on ws://127.0.0.1:38555/devtools/browser/a87775a3-e108-4561-b1af-f8dd986552e7[0911/084701.157394:ERROR:zygote_host_impl_linux.cc(273)] Failed to adjust OOM score of renderer with pid 1245: Permission denied (13)[0911/084701.162380:WARNING:bluez_dbus_manager.cc(247)] Floss manager not present, cannot set Floss enable/disable.[0911/084701.171288:ERROR:angle_platform_impl.cc(44)] RendererVk.cpp:168 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surfaceERR: RendererVk.cpp:168 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surface[0911/084701.192333:ERROR:angle_platform_impl.cc(44)] RendererVk.cpp:168 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surfaceERR: RendererVk.cpp:168 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surface[0911/084701.192447:ERROR:angle_platform_impl.cc(44)] Display.cpp:1063 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/RendererVk.cpp, enableInstanceExtensions:1619.ERR: Display.cpp:1063 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/RendererVk.cpp, enableInstanceExtensions:1619.[0911/084701.192518:ERROR:gl_display.cc(520)] EGL Driver message (Critical) eglInitialize: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/RendererVk.cpp, enableInstanceExtensions:1619.[0911/084701.192603:ERROR:gl_display.cc(790)] eglInitialize SwANGLE failed with error EGL_NOT_INITIALIZED[0911/084701.192677:ERROR:gl_display.cc(824)] Initialization of all EGL display types failed.[0911/084701.192756:ERROR:gl_ozone_egl.cc(26)] GLDisplayEGL::Initialize failed.[0911/084701.194093:ERROR:zygote_host_impl_linux.cc(273)] Failed to adjust OOM score of renderer with pid 1251: Permission denied (13)[0911/084701.194170:ERROR:angle_platform_impl.cc(44)] RendererVk.cpp:168 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surfaceERR: RendererVk.cpp:168 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surface[0911/084701.194300:ERROR:angle_platform_impl.cc(44)] RendererVk.cpp:168 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surfaceERR: RendererVk.cpp:168 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surface[0911/084701.194409:ERROR:angle_platform_impl.cc(44)] Display.cpp:1063 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/RendererVk.cpp, enableInstanceExtensions:1619.ERR: Display.cpp:1063 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/RendererVk.cpp, enableInstanceExtensions:1619.[0911/084701.194675:ERROR:gl_display.cc(520)] EGL Driver message (Critical) eglInitialize: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/RendererVk.cpp, enableInstanceExtensions:1619.[0911/084701.194770:ERROR:gl_display.cc(790)] eglInitialize SwANGLE failed with error EGL_NOT_INITIALIZED[0911/084701.194831:ERROR:gl_display.cc(824)] Initialization of all EGL display types failed.[0911/084701.194914:ERROR:gl_ozone_egl.cc(26)] GLDisplayEGL::Initialize failed.[0911/084701.197338:ERROR:zygote_host_impl_linux.cc(273)] Failed to adjust OOM score of renderer with pid 1256: Permission denied (13)[0911/084701.198668:ERROR:viz_main_impl.cc(186)] Exiting GPU process due to errors during initialization[0911/084701.209657:ERROR:zygote_host_impl_linux.cc(273)] Failed to adjust OOM score of renderer with pid 1272: Permission denied (13)

From my research this is fine to ignore, rendering if also working because you can request via the share options of the panel.

georgerobinson · September 11, 2023, 10:52am

You should also see debug logs from Slack too. For example:

DEBUG[09-11|11:50:23] Creating slack message                   logger=ngalert.notifier.slack
DEBUG[09-11|11:50:23] Synchronizing Alertmanagers for orgs     logger=ngalert.multiorg.alertmanager
DEBUG[09-11|11:50:23] sending Slack API request                logger=ngalert.notifier.slack
DEBUG[09-11|11:50:23] Looks like we are using the Slack API, have set the Bearer token for this request logger=ngalert.notifier.slack
DEBUG[09-11|11:50:23] The request was successful               logger=ngalert.notifier.slack
DEBUG[09-11|11:50:23] Received an image token in annotations   logger=ngalert.notifier.image-provider
DEBUG[09-11|11:50:23] Uploadimg image                          logger=ngalert.notifier.slack
DEBUG[09-11|11:50:23] Sending multipart request to files.upload logger=ngalert.notifier.slack
DEBUG[09-11|11:50:24] The request was successful               logger=ngalert.notifier.slack
DEBUG[09-11|11:50:24] Notify success                           logger=ngalert.notifier.alertmanager LOG15_ERROR= component=alertmanager orgID=1 component=dispatcher receiver=Slack integration=slack[0] aggrGroup={}:{} alerts="[Panel Title[dd21459][active]]" attempts=1

kasiagogolek · September 19, 2023, 11:45am

Hey apologies for the delay in reply - I work with Luke and took over the ticket to setup the images.
We’re not seeing any logs that would suggest that slack is trying to send an image - I can see the output of the slack request and it doesn’t include any information on it attempting sending the image token or url.
the image however is generated by grafana-image-renderer.
If I can give some more info on our setup:

we are using the bot token
bot has access to file:write
we set the image to be saved in local storage (we don’t want to maintain public S3 buckets)
generating images in grafana works fine

One thing I was wondering in this is whether the fact our grafana is on EKS could be affecting the slack notifier being able to read the image? how does the slack notifier get the image?

georgerobinson · September 19, 2023, 1:35pm

Hi! Your set up sounds about right to me. What logs do you see for ngalert.notifier.slack?

kasiagogolek · September 19, 2023, 4:22pm

this is probably one request when we send it:

2023-09-19 17:08:55.739	
logger=ngalert.notifier.slack notifierUID=1ExGyWi4k t=2023-09-19T16:08:55.73923179Z level=debug msg="The request was successful"


2023-09-19 17:08:55.564	
logger=ngalert.notifier.slack notifierUID=1ExGyWi4k t=2023-09-19T16:08:55.564754913Z level=debug msg="Looks like we are using the Slack API, have set the Bearer token for this request"
2023-09-19 17:08:55.564	
logger=ngalert.notifier.slack notifierUID=1ExGyWi4k t=2023-09-19T16:08:55.56473361Z level=debug msg="sending Slack API request" url=https://slack.com/api/chat.postMessage data="{\"channel\":\"#incidents-test\",\"username\":\"bumbum\",\"attachments\":[{\"title\":\"[FIRING:1]  (TestAlert bum testing Grafana)\",\"title_link\":\"https://grafana.sportscape.pro/alerting/list\",\"text\":\"**Firing**\\n\\nValue: [no value]\\nLabels:\\n - alertname = TestAlert\\n - Group = bum\\n - Service = testing\\n - instance = Grafana\\nAnnotations:\\n - summary = Notification test\\nSilence: https://grafana.sportscape.pro/alerting/silence/new?alertmanager=grafana\\u0026matcher=Group%3Dbum\\u0026matcher=Service%3Dtesting\\u0026matcher=alertname%3DTestAlert\\u0026matcher=instance%3DGrafana\\nDashboard: https://grafana.sportscape.pro/d/2RYMRy8nk\\nPanel: https://grafana.sportscape.pro/d/2RYMRy8nk?viewPanel=26\\n\",\"fallback\":\"[FIRING:1]  (TestAlert bum testing Grafana)\",\"footer\":\"Grafana v10.0.2\",\"footer_icon\":\"https://grafana.com/static/assets/img/fav32.png\",\"color\":\"#D63232\",\"ts\":1695139735,\"pretext\":\"\\u003c@@kgogolek\\u003e\",\"mrkdwn_in\":[\"pretext\"]}]}"
2023-09-19 17:08:55.564	
logger=ngalert.notifier.slack notifierUID=1ExGyWi4k t=2023-09-19T16:08:55.564410133Z level=error msg="Missing group labels"
2023-09-19 17:08:55.564	
logger=ngalert.notifier.slack notifierUID=1ExGyWi4k t=2023-09-19T16:08:55.564390787Z level=error msg="Missing receiver"
2023-09-19 17:08:55.564	
logger=ngalert.notifier.slack notifierUID=1ExGyWi4k t=2023-09-19T16:08:55.564370146Z level=debug msg="Creating slack message" alerts=1

I think potentially the issue might be the Missing receiver and Missing group labels which throw an error blocking the slack notifier from sending the follow up image based on this code:
https://github.com/grafana/grafana/blob/v10.0.2/pkg/services/alerting/notifiers/slack.go:317

if err := sn.sendRequest(evalContext.Ctx, data); err != nil {
		return err
	}

	if sn.token != "" && sn.UploadImage {
		err := sn.slackFileUpload(evalContext, sn.log, sn.recipient, sn.token)
		if err != nil {
			return err
		}
	}

but I’m not sure where they’re coming from or whether my assumption is correct? I can’t see the errors in the source to try and work out what could cause them my message for slack send is also different from the one you posted

kasiagogolek · September 19, 2023, 5:16pm

So I actually managed to solve it:

the two errors and no image were showing when I was just running contact point test, despite adding dashboard and panel - that’s ok however confused some results.

in the end I got it working with “real” alerts by adding “uploadImage”: true to the contact point configuration using the admin panel

do you know if there’s a plan for this to be an option on contact point configuration? I only managed to work it out by going through the source code of slack notifier

georgerobinson · September 19, 2023, 6:10pm

Hi!

Yes, images don’t show in contact point testing. We are aware of this limitation. I didn’t realize you were using the Test button. Apologies, otherwise I could have provided the answer earlier!

Btw there is no "uploadImage" option in Alertmanager configuration, that’s just a no-op (is ignored).

kasiagogolek · September 20, 2023, 11:56am

it’s definitely that option that fixed it for us so I think it’s defo used there. Might be something specific about our setup but it doesn’t seem like that option is by default set to true and without that this code in notifier won’t trigger

	if sn.token != "" && sn.UploadImage {
		err := sn.slackFileUpload(evalContext, sn.log, sn.recipient, sn.token)
		if err != nil {
			return err
		}
	}

hopefully this might help someone else who struggles with the notifier setup

georgerobinson · September 20, 2023, 12:03pm

Hi! No, I can promise you that option does nothing in Grafana Alerting. We never read the uploadImage option in Grafana Alerting. That option, along with the code you linked is from Legacy Alerting which you are not using and will be removed in Grafana 11.