"user token not found"

Grafana Configuration
, ,
1

Hi,

We are using a small grafana setup in out k8s in high availability mode (2 instances).
It runs just fine until we recreate the grafana pod (for updates or whatever) the we get the following error:

2022-07-12T06:02:28.694723349Z stdout F logger=context traceID=00000000000000000000000000000000 userId=0 orgId=0 uname= t=2022-07-12T06:02:28.694618932Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=xxx.xxx.xxx.xxx time_ms=0 duration=788.94µs size=27 referer= traceID=00000000000000000000000000000000
2022-07-12T06:02:28.694225421Z stdout F logger=context traceID=00000000000000000000000000000000 t=2022-07-12T06:02:28.694106924Z level=error msg="Failed to look up user based on cookie" error="user token not found"

I assume the issue is, that we use sqlite and the user tokens get lost when creating a new grafana pod?
am i correct?
is there a workaround to make it work without setting up a postgres or mysql server? :slight_smile:

Helmfile snippet, if you need more informations 
  - name: kube-prometheus
    namespace: monitoring
    chart: prometheus-community/kube-prometheus-stack
    version: 36.2.1
    disableValidationOnInstall: true
    values:
      - grafana:
          replicas: 2
          # API access needed for dashboards
          podLabels:
            api-access: "true"
            aadpodidbinding: monitoring-secret-provider-identity
          affinity:
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
                - weight: 1
                  podAffinityTerm:
                    topologyKey: kubernetes.io/hostname
                    labelSelector:
                      matchLabels:
                        app.kubernetes.io/name: grafana
                        app.kubernetes.io/instance: kube-prometheus
          ingress:
            enabled: true
            ingressClassName: global-nginx
            annotations:
              # Session stickyness is required for high availability
              nginx.ingress.kubernetes.io/affinity: "cookie"
            hosts:
              - grafana.{{ requiredEnv "DOMAIN" }}
            tls:
              - hosts:
                  - grafana.{{ requiredEnv "DOMAIN" }}
          persistence:
            enabled: false
          datasources:
            datasources.yaml:
              apiVersion: 1
              datasources:
                - name: Loki
                  type: loki
                  url: http://loki.loki.svc:3100
                  isDefault: false
          dashboardProviders:
            dashboardproviders.yaml:
              apiVersion: 1
              providers:
                - name: default
                  orgId: 1
                  folder: ""
                  type: file
                  disableDeletion: false
                  editable: true
                  options:
                    path: /var/lib/grafana/dashboards/default
          dashboards:
            default:
              external-dns:
                gnetId: 15038
                revision: 1
                datasource: Prometheus
              calico:
                gnetId: 12175
                revision: 5
                datasource: Prometheus
              loki:
                gnetId: 13407
                revision: 1
                datasource: Prometheus
              ingress-nginx:
                gnetId: 9614
                revision: 1
                datasource: Prometheus
              node-exporter-full:
                gnetId: 1860
                revision: 23
                datasource: Prometheus
              vpa-recommendations:
                gnetId: 14588
                revision: 2
                datasource: Prometheus
              jaeger:
                url: https://raw.githubusercontent.com/jaegertracing/jaeger/master/monitoring/jaeger-mixin/dashboard-for-grafana.json
              cert-manager-dashboard:
                gnetId: 11001
                revision: 1
                datasource: Prometheus
              elasticsearch:
                gnetId: 14191
                revision: 1
                datasource: Prometheus

Thanks

2

I don’t know if the grafana DB can be sqlite in HA mode…

3

@mahagon We are getting the same errors while using postgres as a statefulset. Can I know how this error is solved for you.

4

Hi, we weren’t using postgresql at the time.
The solution was using a postgres i’m afraid :confused:

we also added stickyness to the nginx nginx.ingress.kubernetes.io/affinity: “cookie”
ingress-nginx/annotations.md at main · kubernetes/ingress-nginx · GitHub

it works fine since then

1 Like