Losing grafana configurations on K8s when the control plane is rebooted

Grafana Configuration
1
  • What Grafana version and what operating system are you using?

v10.0.3

  • What are you trying to achieve?

having grafana login, prometheus connector and dashboard imports persist.

  • How are you trying to achieve it?

We have grafana configured to use a persistent volume claim and persistent volumes.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: graf-data
  namespace: monitoring
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  selector:
    matchLabels:
      name: nfs-pv-graf
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      name: grafana
      labels:
        app: grafana
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      containers:
      - name: grafana
        image: grafana/grafana:latest
        ports:
        - name: grafana
          containerPort: 3000
        resources:
          limits:
            memory: "1Gi"
            cpu: "1000m"
          requests:
            memory: 500M
            cpu: "500m"
        volumeMounts:
          - mountPath: /grafana/
            name: grafana-storage
          - mountPath: /etc/grafana/provisioning/datasources
            name: grafana-datasources
            readOnly: false
      volumes:
        - name: grafana-storage
          persistentVolumeClaim:
              claimName: graf-data
        - name: grafana-datasources
          configMap:
              defaultMode: 420
              name: grafana-datasources
  • What happened?

If we have a cluster reboot or control plane reboot, the data (login, dashboard configurations ) are gone.

  • What did you expect to happen?

for the login, dashboard configurations to persist.

  • Can you copy/paste the configuration(s) that you are having problems with?

kubernetes 1.27.3
Centos 7
Grafana 10.0.3

2

See Configure a Grafana Docker image | Grafana documentation
Default data path is /var/lib/grafana, but you are mounting volume to /grafana/ → data in /var/lib/grafana are not persistent.

3

Im confused. I have

    volumeMounts:
      - mountPath: /grafana/
        name: grafana-storage
      - mountPath: /etc/grafana/provisioning/datasources
        name: grafana-datasources
        readOnly: false
  volumes:
    - name: grafana-storage
      persistentVolumeClaim:
          claimName: graf-data

i was interpreting that has /grafana in the pod was mounted to persistent volume in the persistentVolumeClaim named graf-data

graf-data is bound to a pv nfs-pv-graf

NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
nfs-pv-am 2Gi RWO Recycle Bound monitoring/am-data 40d
nfs-pv-graf 2Gi RWO Recycle Bound monitoring/graf-data 40d
nfs-pv-prom 2Gi RWO Recycle Bound monitoring/prom-data 40d

what am i missing?

4

Correct, but where did you specified that Grafana has to use /grafana for data? I don’t see that config, so default /var/lib/grafana is used instead for data.

5

So are you saying that I need to retrofit the deploy to have this kind of configuration?

    volumeMounts:
      - mountPath: /grafana/
        name: grafana-storage
     -  mountPath: /var/lib/grafana
        name: grafana-storage
      - mountPath: /etc/grafana/provisioning/datasources
        name: grafana-datasources
        readOnly: false
  volumes:
    - name: grafana-storage
      persistentVolumeClaim:
          claimName: graf-data

I tried the above and I am not seeing the data in the PV

I also tried this

    volumeMounts:
     -  mountPath: /var/lib/grafana
        name: grafana-storage
      - mountPath: /etc/grafana/provisioning/datasources
        name: grafana-datasources
        readOnly: false
  volumes:
    - name: grafana-storage
      persistentVolumeClaim:
          claimName: graf-data

and still no data is stored in the PV

6

I don’t know. Check on the container level, which files have been changed.

BTW 1: Serious Grafana deployments use serious DB (PostgresSQL, MySQL). Default sqlite is good for quick simple testing, but you may expect problems in the future - especially if you have concurent write access.

BTW 2: there is official helm chart GitHub - grafana/helm-charts - you may save some troubles by using it