Grafana - Kubernetes / NGINX Ingress Controller

Looking at Centralized Log Monitoring for our Kubernetes Cluster deployed in Azure AKS, and came across, Grafana I see lot of potential and hope to prototype something and showcase its potential to folks that make decisions.

In our use case we haveNGINX Ingress Controller running on port 443 and which will then route to Grafana Dashboard.

Most Grafana/Kubernetes deployments do not have clear direction of how to achieve this.

When I map this to NGINX Ingress, I run into TOO_MANY_REDIRECT errors, I can’t even login to the Grafana dashboard.

Minikube Configuration

  • NGINX Ingress Controller
  • Grafana YAML for deployment
  • Grafana ini Config Map
  • Grafana Ingress

All seem to start well, but then get TOO_MANY_REDIRECT errors and HTTP errors code 302

ConfigMap - grafana.ini

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-ini
  namespace: grafana
data:
  grafana.ini: |
    [server]
      domain = grafana.malcolmpereira.com
      root_url: https://grafana.malcolmpereira.com
      serve_from_sub_path: false

Grafana yaml

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: grafana
  name: grafana
spec:
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      securityContext:
        fsGroup: 472
        supplementalGroups:
          - 0
      containers:
        - name: grafana
          image: grafana/grafana:8.4.4
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000
              name: http-grafana
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /robots.txt
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 2
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 3000
            timeoutSeconds: 1
          resources:
            requests:
              cpu: 250m
              memory: 750Mi
            limits:
              cpu: 750m
              memory: 1048Mi
          volumeMounts:
            - name: grafana-pv
              mountPath: /var/lib/grafana
            - name: grafana-config
              mountPath: /etc/grafana
      volumes:
        - name: grafana-pv
          persistentVolumeClaim:
            claimName: grafana-pvc
        - name: grafana-config
          configMap:
            name: grafana-ini
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
spec:
  ports:
    - port: 3000
      protocol: TCP
      targetPort: http-grafana
  selector:
    app: grafana
  sessionAffinity: None

Grafana Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/server-snippets: |
      map $http_upgrade $connection_upgrade {
          default upgrade;
          '' close;
      }
      server {
        listen 80;
        root /usr/share/nginx/html;
        index index.html index.htm;
        
        location / {
          proxy_set_header Host $http_host;
          proxy_pass http://grafana.malcolmpereira.com:3000/;
        }
        location /api/live {
          rewrite  ^/(.*)  /$1 break;
          proxy_http_version 1.1;
          proxy_set_header Upgrade $http_upgrade;
          proxy_set_header Connection $connection_upgrade;
          proxy_set_header Host $http_host;
          proxy_pass http://grafana.malcolmpereira.com:3000/;
          }
      }
spec:
  rules:
    - host: grafana.malcolmpereira.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: grafana
                port:
                  number: 3000

Every this seems to start well, but then there are 302 errors and too many redirect errors in the logs.

.3 time_ms=0 size=29 referer=
logger=context t=2022-04-05T03:40:56.28+0000 lvl=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.65.3 time_ms=0 size=29 referer=
logger=context t=2022-04-05T03:40:56.29+0000 lvl=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.65.3 time_ms=0 size=29 referer=
logger=context t=2022-04-05T03:40:56.3+0000 lvl=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.65.3 time_ms=0 size=29 referer=
logger=context t=2022-04-05T03:40:56.32+0000 lvl=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.65.3 time_ms=0 size=29 referer=
logger=context t=2022-04-05T03:40:56.33+0000 lvl=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.65.3 time_ms=0 size=29 referer=
logger=context t=2022-04-05T03:40:56.35+0000 lvl=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.65.3 time_ms=0 size=29 referer=
logger=context t=2022-04-05T03:40:56.36+0000 lvl=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.65.3 time_ms=0 size=29 referer=
logger=context t=2022-04-05T03:40:56.37+0000 lvl=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.65.3 time_ms=0 size=29 referer=
logger=context t=2022-04-05T03:40:56.39+0000 lvl=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.65.3 time_ms=0 size=29 referer=
logger=context t=2022-04-05T03:40:56.4+0000 lvl=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.65.3 time_ms=0 size=29 referer=
logger=context t=2022-04-05T03:40:56.41+0000 lvl=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.65.3 time_ms=0 size=29 referer

Any pointers would be appreciated

I looked at Run Grafana behind a reverse proxy | Grafana Labs, not very clear for my use case.

Did some more reading and could figure this out