Couldn't access Grafana via Nginx Reverse Proxy

Grafana
1

Hey guys,

I have installed Grafana (grafana/grafana:7.2.0) via Helm with additional settings.

This is the additional settings

persistence:
  enabled: true
  persistentVolume:
    storageClass: standard

datasources:
 datasources.yaml:
   apiVersion: 1
   datasources:
   - name: Prometheus
     type: prometheus
     url: http://prometheus-server
     access: proxy
     isDefault: true

This is how I installed

helm install grafana grafana/grafana -f grafana-values.yml --namespace monitoring

The pod is up and running. I tried port forwarding the Pod to local and I can access it via local browser just fine.

kubectl port-forward <grafana pod name> 3000
# logs
t=2020-10-12T07:33:49+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=302 remote_addr=127.0.0.1 time_ms=0 size=29 referer=
t=2020-10-12T07:34:00+0000 lvl=info msg="Successful Login" logger=http.server User=admin@localhost

Now, I tried putting Nginx Pod in front and reverse proxy back to the Grafana Service. The Nginx config

server {
    listen [::]:80;
    listen 80;
    server_name grafana.example.com;
    client_max_body_size 50M;
    large_client_header_buffers 4 32k;

    location / {
        include /etc/nginx/sites-enabled/toolings/auth.conf; # basic auth
        proxy_pass http://grafana.monitoring.svc.cluster.local;
    }
}

However, when I go to grafana.example.com in the browser, I get this message

{"message":"Invalid username or password"}

this is the log

t=2020-10-12T07:34:38+0000 lvl=eror msg="Invalid username or password" logger=context error="Invalid Username or Password"
t=2020-10-12T07:34:38+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=401 remote_addr=10.148.0.177 time_ms=0 size=42 referer=

Anything that I missed? Or any other info that I need to provide?

Edit:

I have tried resetting the password via kubectl as well

kubectl exec -it -n monitoring <pod_name> -- /bin/sh -c "/usr/share/grafana/bin/grafana-cli admin reset-admin-password test_password"

After that, I can log in with the new password when I port-forward the Pod but still the same issue when I tried to use it via Nginx reverse proxy.

Thanks, guys

2
3

Nice. Let me check out the guide and make the changes!

4

After I set the root_url to the domain I am serving, it is still not working.
this is the config btw.

$ kubectl get configmap grafana -o yaml
apiVersion: v1
data:
  datasources.yaml: |
    apiVersion: 1
    datasources:
    - access: proxy
      isDefault: true
      name: Prometheus
      type: prometheus
      url: http://prometheus-server
  grafana.ini: |
    [analytics]
    check_for_updates = true
    [grafana_net]
    url = https://grafana.net
    [log]
    mode = console
    [paths]
    data = /var/lib/grafana/data
    logs = /var/log/grafana
    plugins = /var/lib/grafana/plugins
    provisioning = /etc/grafana/provisioning
    [server]
    root_url = http://grafana.<my_domain>.com
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: grafana
    meta.helm.sh/release-namespace: monitoring
  creationTimestamp: "2020-10-10T02:06:04Z"
  labels:
    app.kubernetes.io/instance: grafana
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: grafana
    app.kubernetes.io/version: 7.2.0
    helm.sh/chart: grafana-5.7.3
  name: grafana
  namespace: monitoring
  resourceVersion: "37267341"
  selfLink: /api/v1/namespaces/monitoring/configmaps/grafana
  uid: 6b59d782-e91d-4a6d-81fb-89dd57fcd419
5

Please set the url as same as the nginx server name

try to change the seting in [server] part

root_url = http://grafana.example.com

Also check this URL:

http://grafana.monitoring.svc.cluster.local

it should reply with the grafana page, if you’re using standard port of grafana, change proxy_pass to proper URL

 location / {
        include /etc/nginx/sites-enabled/toolings/auth.conf; # basic auth
        proxy_pass http://grafana.monitoring.svc.cluster.local:3000;
    }

Because you use:

kubectl port-forward <grafana pod name> 3000
6

After I removed the basic auth line from the nginx, it seems to be working now.

# I removed this line
include /etc/nginx/sites-enabled/toolings/auth.conf; # basic auth