Grafana version update is failing liveliness probe

Grafana Installation
1

Grafana pod is failing the liveliness probe after upgrading to version. 8.1.0 from 6.6.1.

Pod is in CrashLoopBackOff state

2

Hi @chayshakt

Im curious: how are you deploying Grafana to k8s?

3

Warning BackOff 4m23s (x3570 over 12h) kubelet Back-off restarting failed container

grafana-sc-dashboard:
Container ID: xxxxx
Image: kiwigrid/k8s-sidecar:1.12.2
Image ID: xxxxxx
Port:
Host Port:
State: Running
Started: Thu, 09 Sep 2021 12:04:47 -0400
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 1Gi
Requests:
cpu: 100m
memory: 256Mi
Environment:
METHOD: WATCH
LABEL: grafana_dashboard
FOLDER: /tmp/dashboards
NAMESPACE: ALL
Mounts:
/tmp/dashboards from sc-dashboard-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from grafana-token-vlr8z (ro)
grafana:
Container ID: containerd:xxxxxxxx
Image: grafana/grafana:8.1.0
Ports: 80/TCP, 3000/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 139
Started: Thu, 09 Sep 2021 12:26:16 -0400
Finished: Thu, 09 Sep 2021 12:26:17 -0400
Ready: False
Restart Count: 9
Limits:
cpu: 2
memory: 2Gi
Requests:
cpu: 100m
memory: 256Mi
Liveness: http-get https://:3000/api/health delay=60s timeout=30s period=10s #success=1 #failure=10
Readiness: http-get https://:3000/api/health delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
GF_SECURITY_ADMIN_USER: <set to the key ‘admin-user’ in secret ‘grafana’> Optional: false
GF_SECURITY_ADMIN_PASSWORD: <set to the key ‘admin-password’ in secret ‘grafana’> Optional: false
TLS_CA_CERT: <set to the key ‘tls.ca’ in secret ‘ca-bundle’> Optional: false
TLS_CLIENT_CERT: <set to the key ‘tls.crt’ in secret ‘client-tls’> Optional: false
TLS_CLIENT_KEY: <set to the key ‘tls.key’ in secret ‘client-tls’> Optional: false
GF_PATHS_DATA: /var/lib/grafana/
GF_PATHS_LOGS: /var/log/grafana
GF_PATHS_PLUGINS: /var/lib/grafana/plugins
GF_PATHS_PROVISIONING: /etc/grafana/provisioning
Mounts:
/etc/grafana/grafana.ini from config (rw,path=“grafana.ini”)
/etc/grafana/provisioning/dashboards/sc-dashboardproviders.yaml from sc-dashboard-provider (rw,path=“provider.yaml”)
/etc/grafana/provisioning/datasources from sc-datasources-volume (rw)
/etc/grafana/provisioning/notifiers/notifiers.yaml from notifier (rw,path=“notifiers.yaml”)
/etc/grafana/tls from tls (rw)
/tmp/dashboards from sc-dashboard-volume (rw)
/var/lib/grafana from storage (rw)
/var/run/secrets/kubernetes.io/serviceaccount from grafana-token-vlr8z (ro)

4

Not even able to see the logs for grafana container

kubectl logs grafanaxxx -c grafana -n xxxxx