Grafana helm chart fails to provision dashboards

jski185 · April 22, 2025, 8:46pm

What Grafana version and what operating system are you using?
Kubernetes (via k3s) version v1.31.6+k3s1
Grafana helm chart version 8.12.1 (which deploys appver 11.6.0)
What are you trying to achieve?
I am trying to deploy stateless grafana with pre-provisioned dashboards from grafana
How are you trying to achieve it?
Using a custom values file from the grafana helm charts. Here’s my section on provisioning:

      dashboards:
        default:
          node-dashboard:
            #url: 'https://grafana.com./api/dashboards/1860/revisions/37/download'
            gnetId: 1860
            revision: 37
            datasource: Prometheus
      dashboardProviders:
        dashboardproviders.yaml:
          apiVersion: 1
          providers:
          - name: 'default'
            orgId: 1
            folder: ''
            type: file
            disableDeletion: false
            editable: true
            options:
              path: /var/lib/grafana/dashboards/default

What happened?
I see the following code from the download-dashboards sidecar container sidecar, but I do not see the dashboard in the grafana dashboards page. When I shell into the container running grafana, the file /var/lib/grafana/dashboards/default exists but is blank.

+ set -euf                                                                                                                                                                                                                                + mkdir -p /var/lib/grafana/dashboards/default                                                                                                                                                                                         
+ curl -skf --connect-timeout 60 --max-time 60 -H 'Accept: application/json' -H 'Content-Type: application/json;charset=UTF-8' https://grafana.com/api/dashboards/1860/revisions/37/download                                             
+ sed '/-- .* --/! s/"datasource":.*,/"datasource": "Prometheus",/g'                                                                                                                                                                     
stream closed EOF for grafana/grafana-7b8d758c49-m9wh7 (download-dashboards)

What did you expect to happen?
I expected the dashboards are loaded when I open the page in my grafana instance.
Can you copy/paste the configuration(s) that you are having problems with?
See above.
Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
Repeating error below:
logger=provisioning.dashboard type=file name=default t=2025-04-22T20:44:36.591412407Z level=error msg="failed to load dashboard from " file=/var/lib/grafana/dashboards/default/node-dashboard.json error=EOF
Did you follow any online instructions? If so, what is the URL?
I used the documentation on the grafana helm charts repo.

shawnpitts · November 12, 2025, 12:03am

Hi @jski185, my names Shawn and I met you today at KubeCON! Since i’m running this locally on a Macbook I used k3d so I could run it using docker instead of k3s like you have but I was successfully able to import the “Node Exporter Full” dashboard with the following values.yaml

dashboards:
  default:
    node-dashboard:
      url: 'https://grafana.com/api/dashboards/1860/revisions/37/download'
      gnetId: 1860
      revision: 37
      datasource: Prometheus
dashboardProviders:
  dashboardproviders.yaml:
    apiVersion: 1
    providers:
    - name: 'default'
      orgId: 1
      folder: ''
      type: file
      disableDeletion: false
      editable: true
      options:
        path: /var/lib/grafana/dashboards/default

All I needed to do was

$ helm install my-grafana grafana/grafana --version 8.12.1 -f values.yaml

then port forward the grafana svc

$ kubectl port-forward svc/my-grafana 3000:80

Once that got up and running I navigated to dashboards and see the following.

jski185 · November 12, 2025, 8:24pm

Hey Shawn, I just copied it into my chart values.yaml and it’s still not wanting to work. I will try and swing by your booth if you’ll still be there tomorrow? I will bring my laptop so I can show you my setup. Cheers!

jski185 · November 14, 2025, 3:16am

Here are those logs we were talking about.

From the download-dashboards sidecar:

+ set -euf
+ mkdir -p /var/lib/grafana/dashboards/default
+ curl -skf --connect-timeout 60 --max-time 60 -H 'Accept: application/json' -H 'Content-Type: application/json;charset=UTF-8' https://grafana.com/api/dashboards/1860/revisions/37/download
+ sed '/-- .* --/! s/"datasource":.*,/"datasource": "Prometheus",/g'
stream closed EOF for grafana/grafana-6cbb46cf65-s49cq (download-dashboards)

For the grafana pod after startup:

logger=provisioning.dashboard type=file name=default t=2025-11-14T03:09:50.24021762Z level=error msg="failed to load dashboard from " file=/var/lib/grafana/dashboards/default/node-dashboard.json error=EOF
logger=provisioning.dashboard type=file name=default t=2025-11-14T03:10:00.241296784Z level=error msg="failed to load dashboard from " file=/var/lib/grafana/dashboards/default/node-dashboard.json error=EOF
logger=provisioning.dashboard type=file name=default t=2025-11-14T03:10:10.24039441Z level=error msg="failed to load dashboard from " file=/var/lib/grafana/dashboards/default/node-dashboard.json error=EOF
logger=provisioning.dashboard type=file name=default t=2025-11-14T03:10:20.24061105Z level=error msg="failed to load dashboard from " file=/var/lib/grafana/dashboards/default/node-dashboard.json error=EOF
logger=provisioning.dashboard type=file name=default t=2025-11-14T03:10:30.240681199Z level=error msg="failed to load dashboard from " file=/var/lib/grafana/dashboards/default/node-dashboard.json error=EOF
logger=provisioning.dashboard type=file name=default t=2025-11-14T03:10:40.24057808Z level=error msg="failed to load dashboard from " file=/var/lib/grafana/dashboards/default/node-dashboard.json error=EOF
logger=provisioning.dashboard type=file name=default t=2025-11-14T03:10:50.240744992Z level=error msg="failed to load dashboard from " file=/var/lib/grafana/dashboards/default/node-dashboard.json error=EOF
logger=provisioning.dashboard type=file name=default t=2025-11-14T03:11:00.241045073Z level=error msg="failed to load dashboard from " file=/var/lib/grafana/dashboards/default/node-dashboard.json error=EOF
logger=infra.usagestats t=2025-11-14T03:11:03.96475089Z level=info msg="Usage stats are ready to report"
logger=provisioning.dashboard type=file name=default t=2025-11-14T03:11:10.241013982Z level=error msg="failed to load dashboard from " file=/var/lib/grafana/dashboards/default/node-dashboard.json error=EOF
logger=provisioning.dashboard type=file name=default t=2025-11-14T03:11:20.240368093Z level=error msg="failed to load dashboard from " file=/var/lib/grafana/dashboards/default/node-dashboard.json error=EOF

Inside the grafana pod, the file referenced above is (as you can tell from the error) a 0-byte empty file.

From the grafana cm (with some data redacted):

Name:         grafana
Namespace:    grafana
Labels:       app.kubernetes.io/instance=grafana
              app.kubernetes.io/name=grafana
              app.kubernetes.io/version=12.1.0
              helm.sh/chart=grafana-9.3.5
Annotations:  argocd.argoproj.io/tracking-id: grafana:/ConfigMap:grafana/grafana

Data
====
dashboardproviders.yaml:
----
apiVersion: 1
providers:
- disableDeletion: false
  editable: true
  folder: ""
  name: default
  options:
    path: /var/lib/grafana/dashboards/default
  orgId: 1
  type: file


datasources.yaml:
----
apiVersion: 1
datasources:
- access: proxy
  isDefault: true
  name: Prometheus
  type: prometheus
  url: http://prometheus-server.prometheus.svc.cluster.local
- access: proxy
  jsonData:
    httpHeaderName1: X-Scope-OrgID
    tlsSkipVerify: true
  name: Loki
  secureJsonData:
    httpHeaderValue1: empty
  type: loki
  url: http://loki-gateway.loki.svc.cluster.local


download_dashboards.sh:
----
#!/usr/bin/env sh
set -euf
mkdir -p /var/lib/grafana/dashboards/default

curl -skf \
--connect-timeout 60 \
--max-time 60 \
-H "Accept: application/json" \
-H "Content-Type: application/json;charset=UTF-8" \
  "https://grafana.com/api/dashboards/1860/revisions/37/download" \
  | sed '/-- .* --/! s/"datasource":.*,/"datasource": "Prometheus",/g' \
> "/var/lib/grafana/dashboards/default/node-dashboard.json"


grafana.ini:
----
[analytics]
check_for_updates = true
[auth]
disable_login_form = true
oauth_auto_login = false
[auth.anonymous]
enabled = false
[auth.generic_oauth]
{redacted}
[grafana_net]
url = https://grafana.net
[log]
mode = console
[paths]
data = /var/lib/grafana/
logs = /var/log/grafana
plugins = /var/lib/grafana/plugins
provisioning = /etc/grafana/provisioning
[security]
disable_initial_admin_creation = true
[server]
{redacted}
[users]
{redacted}

Happy to keep digging into this!

Topic		Replies	Views
Grafana Dashboard Provisioning failing through Helm Grafana provisioning	0	2402	June 10, 2019
How to provision a dashboard from grafana.com/dashboards when deploying Grafana using the Helm chart? Installation	13	12719	November 29, 2023
Grafana provisioning dashboards Grafana dashboard , grafana	0	35	November 17, 2025
Dashboard Provisioning deploying to Kubernetes using Helm charts Dashboards	0	410	November 29, 2023
How to upload dashboards as .json files to kubernetes via helm Installation provisioning , backend-platform , kubernetes , helm , dashboards-as-code	31	58332	October 18, 2024

Grafana helm chart fails to provision dashboards

Related topics