Looking at Centralized Log Monitoring for our Kubernetes Cluster deployed in Azure AKS, and came across, Grafana I see lot of potential and hope to prototype something and showcase its potential to folks that make decisions.
In our use case we haveNGINX Ingress Controller running on port 443 and which will then route to Grafana Dashboard.
Most Grafana/Kubernetes deployments do not have clear direction of how to achieve this.
When I map this to NGINX Ingress, I run into TOO_MANY_REDIRECT errors, I can’t even login to the Grafana dashboard.
Minikube Configuration
- NGINX Ingress Controller
- Grafana YAML for deployment
- Grafana ini Config Map
- Grafana Ingress
All seem to start well, but then get TOO_MANY_REDIRECT errors and HTTP errors code 302
ConfigMap - grafana.ini
---
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-ini
namespace: grafana
data:
grafana.ini: |
[server]
domain = grafana.malcolmpereira.com
root_url: https://grafana.malcolmpereira.com
serve_from_sub_path: false
Grafana yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: grafana-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: grafana
name: grafana
spec:
selector:
matchLabels:
app: grafana
template:
metadata:
labels:
app: grafana
spec:
securityContext:
fsGroup: 472
supplementalGroups:
- 0
containers:
- name: grafana
image: grafana/grafana:8.4.4
imagePullPolicy: IfNotPresent
ports:
- containerPort: 3000
name: http-grafana
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /robots.txt
port: 3000
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 2
livenessProbe:
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 3000
timeoutSeconds: 1
resources:
requests:
cpu: 250m
memory: 750Mi
limits:
cpu: 750m
memory: 1048Mi
volumeMounts:
- name: grafana-pv
mountPath: /var/lib/grafana
- name: grafana-config
mountPath: /etc/grafana
volumes:
- name: grafana-pv
persistentVolumeClaim:
claimName: grafana-pvc
- name: grafana-config
configMap:
name: grafana-ini
---
apiVersion: v1
kind: Service
metadata:
name: grafana
spec:
ports:
- port: 3000
protocol: TCP
targetPort: http-grafana
selector:
app: grafana
sessionAffinity: None
Grafana Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: grafana-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/server-snippets: |
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
server {
listen 80;
root /usr/share/nginx/html;
index index.html index.htm;
location / {
proxy_set_header Host $http_host;
proxy_pass http://grafana.malcolmpereira.com:3000/;
}
location /api/live {
rewrite ^/(.*) /$1 break;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Host $http_host;
proxy_pass http://grafana.malcolmpereira.com:3000/;
}
}
spec:
rules:
- host: grafana.malcolmpereira.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: grafana
port:
number: 3000
Every this seems to start well, but then there are 302 errors and too many redirect errors in the logs.
.3 time_ms=0 size=29 referer=
logger=context t=2022-04-05T03:40:56.28+0000 lvl=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.65.3 time_ms=0 size=29 referer=
logger=context t=2022-04-05T03:40:56.29+0000 lvl=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.65.3 time_ms=0 size=29 referer=
logger=context t=2022-04-05T03:40:56.3+0000 lvl=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.65.3 time_ms=0 size=29 referer=
logger=context t=2022-04-05T03:40:56.32+0000 lvl=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.65.3 time_ms=0 size=29 referer=
logger=context t=2022-04-05T03:40:56.33+0000 lvl=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.65.3 time_ms=0 size=29 referer=
logger=context t=2022-04-05T03:40:56.35+0000 lvl=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.65.3 time_ms=0 size=29 referer=
logger=context t=2022-04-05T03:40:56.36+0000 lvl=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.65.3 time_ms=0 size=29 referer=
logger=context t=2022-04-05T03:40:56.37+0000 lvl=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.65.3 time_ms=0 size=29 referer=
logger=context t=2022-04-05T03:40:56.39+0000 lvl=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.65.3 time_ms=0 size=29 referer=
logger=context t=2022-04-05T03:40:56.4+0000 lvl=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.65.3 time_ms=0 size=29 referer=
logger=context t=2022-04-05T03:40:56.41+0000 lvl=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.65.3 time_ms=0 size=29 referer
Any pointers would be appreciated
I looked at Run Grafana behind a reverse proxy | Grafana Labs, not very clear for my use case.