Grafana image renderer - 401 Unauthorized when clicking "Direct link rendered image"

Hi, please can someone help with this. I have been trying for hours and am unable to make any progress. I have consulted as many forums posts and guides as I can find, but to no avail.

  • What Grafana version and what operating system are you using?
    Windows 11 Pro - 24H2
    Grafana v11.3.0 (d9455ff7db)
    Renderer 3.11.6
    Chrome browser Version 130.0.6723.117 (Official Build) (64-bit)

  • What are you trying to achieve?
    Use Grafana renderer to render my first image

  • How are you trying to achieve it?
    Followed the online documentation and consulted as many community posts as I could

  • What happened?
    When I click “Direct link rendered image” in GUI via Chrome browser I get redirected to
    https://*****.mydomain.com/render/d-solo/ce1l89cw41ds0b?from=2024-11-06T12:00:21.162Z&to=2024-11-06T18:00:21.162Z&timezone=browser&panelId=panel-1&__feature.dashboardSceneSolo&width=1000&height=500&tz=America%2FLos_Angeles

and it shows

{“statusCode”:401,“error”:“Unauthorized”,“message”:“Unauthorized request”}

  • What did you expect to happen?
    Something other than the page I got

  • Can you copy/paste the configuration(s) that you are having problems with?

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: grafana
  name: grafana
  namespace: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      securityContext:
        fsGroup: 472
        supplementalGroups:
          - 0
      containers:
        - name: renderer
          image: grafana/grafana-image-renderer:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8081
              name: http-renderer
              protocol: TCP
          env:
            - name: AUTH_TOKEN
              value: test123
            - name: LOG_LEVEL
              value: debug

        - name: grafana
          image: grafana/grafana:11.3.0
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000
              name: http-grafana
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /robots.txt
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 2
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 3000
            timeoutSeconds: 1
          resources:
            requests:
              cpu: 250m
              memory: 750Mi
          env:
            - name: GF_LOG_FILTERS
              value: rendering:debug
            - name: GF_INSTALL_PLUGINS
              value: *****
            - name: GF_DATABASE_USER
              valueFrom:
                secretKeyRef:
                  name: db-grafana-pass
                  key: username
            - name: GF_DATABASE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-grafana-pass
                  key: password
            - name: GF_AUTH_OKTA_CLIENT_ID
              valueFrom:
                secretKeyRef:
                  name: okta-grafana-pass
                  key: okta_client_id
            - name: GF_AUTH_OKTA_CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  name: okta-grafana-pass
                  key: okta_client_secret
          volumeMounts:
            - mountPath: /var/lib/grafana
              name: grafana-lib-pvc
            - mountPath: /etc/grafana
              name: grafana-config-pvc
      volumes:
        - name: grafana-lib-pvc
          persistentVolumeClaim:
            claimName: grafana-lib-pvc
        - name: grafana-config-pvc
          configMap:
            name: grafana-config
            items:
                - key: grafana.ini
                  path: grafana.ini
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-config
  namespace: grafana
  labels:
    app: grafana
data:
  grafana.ini: |+
      [database]
      type = ****
      host = ****
      name = grafana
      user = GF_DATABASE_USER
      password = GF_DATABASE_PASSWORD

      [log]
      level = info

      [server]
      root_url = https://*****.mydomain.com

      [auth.okta]
      name = Okta
      ... removed ...

      [rendering]
      server_url = http://renderer:80/render
      callback_url = http://grafana:80/
      renderer_token = test123
---
apiVersion: v1
kind: Service
metadata:
  name: renderer
  namespace: grafana
  labels:
    app: renderer
spec:
  type: NodePort
  selector:
    app: grafana
  ports:
  - name: http
    port: 80
    targetPort: http-renderer

---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: grafana
  labels:
    app: grafana
spec:
  type: NodePort
  selector:
    app: grafana
  ports:
  - name: http
    port: 80
    targetPort: http-grafana
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana
  namespace: grafana
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/group.name: ****
    alb.ingress.kubernetes.io/load-balancer-name: *********
    alb.ingress.kubernetes.io/certificate-arn: *********
    alb.ingress.kubernetes.io/backend-protocol: HTTP
    alb.ingress.kubernetes.io/healthcheck-path: /api/health
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/inbound-cidrs: "****"
spec:
  ingressClassName: alb
  rules:
  - http:
      paths:
      - path: /render
        pathType: Prefix
        backend:
          service:
            name: renderer
            port:
              number: 80
      - path: /
        pathType: Prefix
        backend:
          service:
            name: grafana
            port:
              number: 80
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-lib-pvc
  namespace: grafana
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: gp2
  resources:
    requests:
      storage: 1Gi
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
NAME                          READY   STATUS    RESTARTS   AGE
pod/grafana-xxxxxx-xxxx   2/2     Running   0          2m25s

NAME               TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/grafana    NodePort   X.X.X.X    <none>        80:#####/TCP   277d
service/renderer   NodePort   Y.Y.Y.Y   <none>        80:#####/TCP   15d

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/grafana   1/1     1            1           277d
kubectl logs -n grafana -l app=grafana  -c renderer --tail=20
{"level":"debug","message":"::ffff:X.X.X.X - - [06/Nov/2024:17:46:00 +0000] \"GET / HTTP/1.1\" 200 22 \"-\" \"ELB-HealthChecker/2.0\"\n"}
{"level":"debug","message":"::ffff:X.X.X.X - - [06/Nov/2024:17:46:06 +0000] \"GET / HTTP/1.1\" 200 22 \"-\" \"ELB-HealthChecker/2.0\"\n"}
{"level":"error","message":"Request failed","stack":"Error: Unauthorized request\n    at /usr/src/app/build/service/middlewares.js:27:30\n    at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)\n    at trim_prefix (/usr/src/app/node_modules/express/lib/router/index.js:328:13)\n    at /usr/src/app/node_modules/express/lib/router/index.js:286:9\n    at Function.process_params (/usr/src/app/node_modules/express/lib/router/index.js:346:12)\n    at next (/usr/src/app/node_modules/express/lib/router/index.js:280:10)\n    at jsonParser (/usr/src/app/node_modules/body-parser/lib/types/json.js:113:7)\n    at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)\n    at trim_prefix (/usr/src/app/node_modules/express/lib/router/index.js:328:13)\n    at /usr/src/app/node_modules/express/lib/router/index.js:286:9","url":"/render/d-solo/ce1l89cw41ds0b?from=2024-11-06T11:42:53.327Z&to=2024-11-06T17:42:53.327Z&timezone=browser&panelId=panel-1&__feature.dashboardSceneSolo&width=1000&height=500&tz=America%2FLos_Angeles"}
{"level":"error","message":"::ffff:X.X.X.X - - [06/Nov/2024:17:46:07 +0000] \"GET /render/d-solo/ce1l89cw41ds0b?from=2024-11-06T11:42:53.327Z&to=2024-11-06T17:42:53.327Z&timezone=browser&panelId=panel-1&__feature.dashboardSceneSolo&width=1000&height=500&tz=America%2FLos_Angeles HTTP/1.1\" 401 74 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36\"\n"}
{"level":"debug","message":"::ffff:Y.Y.Y.Y - - [06/Nov/2024:17:46:07 +0000] \"GET / HTTP/1.1\" 200 22 \"-\" \"ELB-HealthChecker/2.0\"\n"}
kubectl logs -n grafana -l app=grafana  -c grafana --tail=20
logger=rendering renderer=http t=2024-11-06T17:40:02.153406222Z level=debug msg="calling remote rendering service" url=http://renderer:80/render/version
logger=provisioning.dashboard t=2024-11-06T17:40:02.154274819Z level=error msg="can't read dashboard provisioning files from directory" path=/etc/grafana/provisioning/dashboards error="open /etc/grafana/provisioning/dashboards: no such file or directory"
logger=grafanaStorageLogger t=2024-11-06T17:40:02.155224443Z level=info msg="Storage starting"
logger=ngalert.multiorg.alertmanager t=2024-11-06T17:40:02.15620086Z level=info msg="Starting MultiOrg Alertmanager"
logger=ngalert.state.manager t=2024-11-06T17:40:02.157305569Z level=info msg="State cache has been initialized" states=0 duration=4.894818ms
logger=ngalert.scheduler t=2024-11-06T17:40:02.157379432Z level=info msg="Starting scheduler" tickInterval=10s maxAttempts=1
logger=ticker t=2024-11-06T17:40:02.157798481Z level=info msg=starting first_tick=2024-11-06T17:40:10Z
logger=rendering renderer=http t=2024-11-06T17:40:02.192877897Z level=info msg="Backend rendering via external http server" version=3.11.6
logger=provisioning.dashboard t=2024-11-06T17:40:02.19968561Z level=info msg="starting to provision dashboards"
logger=provisioning.dashboard t=2024-11-06T17:40:02.209657777Z level=info msg="finished to provision dashboards"
logger=plugins.update.checker t=2024-11-06T17:40:02.354486531Z level=info msg="Update check succeeded" duration=198.645159ms
logger=grafana.update.checker t=2024-11-06T17:40:02.46068985Z level=info msg="Update check succeeded" duration=305.470589ms
logger=grafana-apiserver t=2024-11-06T17:40:02.681233887Z level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager"
logger=grafana-apiserver t=2024-11-06T17:40:02.682179039Z level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager"
logger=grafana-apiserver t=2024-11-06T17:40:02.684083758Z level=info msg="Adding GroupVersion iam.grafana.app v0alpha1 to ResourceManager"
logger=infra.usagestats t=2024-11-06T17:40:46.192723601Z level=info msg="Usage stats are ready to report"
logger=context userId=0 orgId=0 uname= t=2024-11-06T17:42:41.068422008Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=A.A.A.A time_ms=1 duration=1.777222ms size=47 referer= handler=/ status_source=server
logger=context userId=0 orgId=0 uname= t=2024-11-06T17:42:41.177394853Z level=info msg="Request Completed" method=GET path=/user/auth-tokens/rotate status=302 remote_addr=A.A.A.A time_ms=4 duration=4.905331ms size=24 referer= handler=/user/auth-tokens/rotate status_source=server
logger=context userId=3 orgId=1 uname=firstname.lastname@mydomain.com t=2024-11-06T17:42:41.975631127Z level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=A.A.A.A time_ms=23 duration=23.54565ms size=0 referer= handler=/api/live/ws status_source=server
logger=live t=2024-11-06T17:42:52.601982351Z level=info msg="Initialized channel handler" channel=grafana/dashboard/uid/ce1l89cw41ds0b address=grafana/dashboard/uid/ce1l89cw41ds0b

Anyone from Grafana able to help with this? I’m still stuck trying to make it work.

Does the image generation works within alerts? I think what you’re missing with manual rendering is telling Grafana how you want to authenticate (I guess your users must be authenticated). Chat GPT says you should add render_key query param with value set to your render key (I think test123).

I don’t use alerts. What would be the easiest way for me to verify whether it works in alerts with a fresh Grafana installation?

I get what you are saying about render_key but that URL is generated from the Grafana GUI not by me. It isn’t clear where this render_key would be specified in the configuration and why it would be in plaintext in the URL. When I inspected the call in developer tools I did not see anything in the header other than the usual Cookie. I am authenticating with Okta SSO to Grafana itself.

Did you solve this? I have 401 unauthorized on image renderer as well. I tried to set environment: AUTH_TOKEN: test123 to both services (grafana and renderer), but it still does not work.

I didn’t solve it :sob:
Still hoping someone has the answer somewhere or a Grafana employee has time to take a look.

If I manually add a header to the browser then I can get past the 404, not exactly a user friendly approach but it is progress.

Add X-Auth-Token=test123

But now I get a 404.
Cannot GET /render/d-solo/dashboard_id

I see a log in the renderer with code 404 167

Anyone know what is going on here?