Grafana stucks on Starting DB migrations

Grafana Configuration
1

I’m trying to configure Grafana on k3s v1.29.1+k3s2 with NFS as PersistentVolume. Grafana correctly connects to NFS server, but container stucks on “Starting DB migrations”

Logs

logger=settings t=2024-02-29T19:49:37.442908249Z level=info msg="Starting Grafana" version=10.3.3 commit=252761264e22ece57204b327f9130d3b44592c01 branch=HEAD compiled=2024-02-29T19:49:37Z
logger=settings t=2024-02-29T19:49:37.443176042Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2024-02-29T19:49:37.443191963Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
logger=settings t=2024-02-29T19:49:37.443200157Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
logger=settings t=2024-02-29T19:49:37.443207974Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
logger=settings t=2024-02-29T19:49:37.443215618Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
logger=settings t=2024-02-29T19:49:37.443223741Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
logger=settings t=2024-02-29T19:49:37.443232428Z level=info msg="Config overridden from command line" arg="default.log.mode=console"
logger=settings t=2024-02-29T19:49:37.443241975Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
logger=settings t=2024-02-29T19:49:37.443249988Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
logger=settings t=2024-02-29T19:49:37.44325752Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
logger=settings t=2024-02-29T19:49:37.443265604Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
logger=settings t=2024-02-29T19:49:37.443273678Z level=info msg=Target target=[all]
logger=settings t=2024-02-29T19:49:37.443293755Z level=info msg="Path Home" path=/usr/share/grafana
logger=settings t=2024-02-29T19:49:37.443303064Z level=info msg="Path Data" path=/var/lib/grafana
logger=settings t=2024-02-29T19:49:37.443310362Z level=info msg="Path Logs" path=/var/log/grafana
logger=settings t=2024-02-29T19:49:37.443318619Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
logger=settings t=2024-02-29T19:49:37.44332657Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
logger=settings t=2024-02-29T19:49:37.443334271Z level=info msg="App mode production"
logger=sqlstore t=2024-02-29T19:49:37.443615487Z level=info msg="Connecting to DB" dbtype=sqlite3
logger=migrator t=2024-02-29T19:49:37.44454707Z level=info msg="Starting DB migrations"

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: grafana
  name: grafana
  namespace: grafana
spec:
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      securityContext:
        fsGroup: 472
        supplementalGroups:
          - 0
      containers:
        - name: grafana
          image: grafana/grafana:10.3.3
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000
              name: http-grafana
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /robots.txt
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 2
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 3000
            timeoutSeconds: 1
          resources:
            requests:
              cpu: 250m
              memory: 750Mi
          volumeMounts:
            - mountPath: /var/lib/grafana
              name: grafana-pv
      volumes:
        - name: grafana-pv
          persistentVolumeClaim:
            claimName: grafana-volume-claim

Volume

apiVersion: v1
kind: PersistentVolume
metadata:
  name: grafana-volume
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""
  mountOptions:
   - hard
   - nfsvers=3
  nfs:
    path: /mnt/data/k8s/grafana
    server: 192.168.1.11

VolumeClaim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-volume-claim
  namespace: grafana
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  volumeName: grafana-volume
  resources:
    requests:
      storage: 1Gi

NFS Configuration

/mnt/data/k8s/grafana   192.168.1.15(rw,all_squash,sync,no_subtree_check)  192.168.1.16(rw,all_squash,sync,no_subtree_check)  192.168.1.18(rw,all_squash,sync,no_subtree_check)
2

Hi,
I have the same problem with Helm installation.
Did you add nodes recently ?
What is the status of your pvc ?

kubectl get pvc -n grafana

Do you see a grafana.db file on your disk with a modification time corresponding to the time of installation ? (I do)

What gives

kub describe pod <your_pod_id> -n grafana

I get:

Warning Unhealthy 9m55s (x121 over 24m) kubelet Readiness probe failed: Get “http://10.2.0.102:3000/api/health”: dial tcp 10.2.0.102:3000: connect: connection refused

3

I bypassed the problem with this