Grafana HA setup errors "failed to save dashboard"

Hi All,

We are running Grafana on kubernetes and recently, we have updated grafana to use postgresDB at the backend. When we increased the replicas for grafana server from 1 to 2 we started seeing this errors in grafana logs:

lvl=eror msg="failed to save dashboard" logger=provisioning.dashboard type=file name=overview error="pq: duplicate key value violates unique constraint \"UQE_dashboard_version_dashboard_id_version\""

We have validated all the dashboards and all of them has unique dashboard uid. Which leaves the suspicion on 2 grafana servers are trying to save same dashboard to database and database giving us this error.

We need help to understand if this is an expected behaviour by design OR are we missing something in our setup.

Deployment config:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:

spec:
progressDeadlineSeconds: 600
replicas: 2
revisionHistoryLimit: 3
selector:

strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 0%
type: RollingUpdate
template:
metadata:
annotations:

creationTimestamp: null
labels:

spec:
containers:
- env:

image: kiwigrid/k8s-sidecar:0.1.178
imagePullPolicy: IfNotPresent
name: dashboards-folder-sidecar
resources:

terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /tmp/dashboards
name: dashboard-volume
- mountPath: /etc/grafana/provisioning/dashboards
name: dashboard-providers
- env:
- name: GF_DATABASE_NAME
value: <GF_DATABASE_NAME>
- name: GF_DATABASE_TYPE
value: postgres
- name: GF_DATABASE_HOST
value: <GF_DATABASE_HOST>
- name: GF_DATABASE_SSL_MODE
value: <GF_DATABASE_SSL_MODE>
envFrom:
- secretRef:
name:
image: grafana/grafana:7.1.5
imagePullPolicy: IfNotPresent
ports:
- containerPort: 3000
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /login
port: 3000
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1

Versions:

  • Grafana version: 7.1.5
  • Kubernetes: 1.15

Please let me know in case more information is required.