[Image-Renderer/S3] Grafana won't upload images to bucket/send images to slack using a webhook

Grafana Alerting
,
1

What Grafana version and what operating system are you using?
Grafana v12.0.2, deployed via Helm chart grafana-9.2.9.
Running on Amazon Linux 2023 (Kubernetes pod).

What are you trying to achieve?
Get alert images from Grafana to display in Slack messages.

How are you trying to achieve it?

  • Using the Grafana Image Renderer sidecar pod.

  • Tested both Slack contact point modes:

    1. Incoming Webhook + external image storage on AWS S3.

    2. Slack bot token (xoxb…) with “Upload image” enabled.

What happened?

  1. Rendering works fine (curl /render/d-solo/... produces PNGs).

  2. In webhook mode, images are uploaded to S3 successfully, but with no consistency, which means they occasionally fail to upload and Slack shows no image.
    Here’s what I tried:

    • Configuring our S3 buckets to block all public access and allow access to Grafana’s K8S Service Account via an IAM role only (org policy).

    • Enabling full public access to the bucket (Still very flaky, sometimes it would work, but with no consistency.

    • Slack can’t fetch the image URL Grafana provides.

  3. In bot token mode: images do get uploaded to Slack, but they appear as threaded replies, not inline with the alert message.

  4. Logs showed a lot of renderer noise at first (timeouts, timezone errors, Prometheus query failures), but I confirmed the renderer is producing temp PNGs (/tmp/xxxx.png) and deleting them after upload.

What did you expect to happen?

  • With webhook + S3: Slack should display the image inline in the alert message.

  • With bot token: Slack should show the image in the main message, not as a reply in a thread.

Can you copy/paste the configuration(s) that you are having problems with?
Here are relevant snippets from our Helm chart values:

env:
  GF_UNIFIED_ALERTING_ENABLED: true
  GF_UNIFIED_ALERTING_SCREENSHOTS_CAPTURE: true
  GF_UNIFIED_ALERTING_SCREENSHOTS_CAPTURE_TIMEOUT: 30s
  GF_EXTERNAL_IMAGE_STORAGE_PROVIDER: s3
  GF_EXTERNAL_IMAGE_STORAGE_S3_BUCKET: grafana-image-rendering-prod
  GF_EXTERNAL_IMAGE_STORAGE_S3_REGION: us-east-1

grafana.ini:
  external_image_storage:
    provider: s3
    s3:
      bucket: grafana-image-rendering-prod
      region: us-east-1
  external_image_storage.s3:
    endpoint: s3.us-east-1.amazonaws.com
    bucket: grafana-image-rendering-prod
    region: us-east-1
  unified_alerting.screenshots:
    capture: true
    capture_timeout: 30s
    max_concurrent_screenshots: 10
    upload_external_image_storage: true

imageRenderer:
  enabled: true
  replicas: 1
  image:
    repository: grafana/grafana-image-renderer
    tag: 3.12.6
  env:
    BROWSER_TZ: Asia/Jerusalem
    RENDERING_ARGS: --no-sandbox,--disable-gpu,--window-size=1280x758
    RENDERING_MODE: reusable

Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
Examples from renderer pod logs:

  • TimeoutError: Timeout error while performing step: panelsRendered

  • Invalid timezone ID: Asia/Jerusalem (fixed by encoding as Asia%2FJerusalem)

  • net::ERR_ABORTED on Prometheus queries during rendering

  • Deleting temporary file /tmp/<hash>.png after each render

Did you follow any online instructions? If so, what is the URL?

**Questions for the community:
**
Why is Grafana together with S3 + webhook so flaky on my end?

  • Sometimes images upload to S3, and sometimes they don’t, with no consistency.

  • What am I missing here?

  • What do I need to make sure of when aiming to achieve said results?

  • For webhook + S3: Is there a supported way for Grafana to generate signed URLs instead of requiring a public bucket?

  • For bot token mode: Is there any way to make the image appear inline in the alert message rather than as a threaded reply?

  • For orgs that must block public access to S3, what’s the recommended approach to reliably send images to Slack?