Unable to login as admin in Grafana

What Grafana version and what operating system are you using?

v10.3.4, Linux

What are you trying to achieve?

We are working on allowing users to access Grafana by port-forwarding from Kubernetes as admin.

How are you trying to achieve it?

I’m running Grafana helm chart version 7.3.9 (latest). I’m running Grafana through terraform and attempting to access by port-forwarding from local command line, eg: kubectl port-forward svc/grafana 3001:80 and logging into localhost:3001 locally.

Some things we’ve tried:

  • I’ve tried running this on Grafana 10.3.4 and 10.2.2.
  • I also tried to run this with Chart version 6.56.1 (we updated the chart to confirm the chart version wasn’t the issue here).
  • I am able to access dashboards/alerts/metrics etc. by setting anonymous auth, but when I set anonymous auth, attempting to log in with username/password still doesn’t work.

What happened?

I’m getting a user/pass failed on attempting to log in.

What did you expect to happen?

I expect to be able to login as admin with admin/password credentials.

Can you copy/paste the configuration(s) that you are having problems with?

Helm chart:

  repository: grafana/grafana
  tag: 10.3.4

adminPassword: password

    apiVersion: 1
    datasources: []

    level: debug
    auto_assign_org_role: Viewer

  enabled: true

  enabled: true
    enabled: true
  logLevel: DEBUG

useStatefulSet: true

This is the grafana.ini from within the pod:

check_for_updates = true
url = https://grafana.net
level = debug
mode = console
data = /var/lib/grafana/
logs = /var/log/grafana
plugins = /var/lib/grafana/plugins
provisioning = /etc/grafana/provisioning
domain = ''
auto_assign_org_role = Viewer

When I SSH into the relevant pod, I see the env variables GF_SECURITY_ADMIN_PASSWORD=password and GF_SECURITY_ADMIN_USER=admin, so admin/password should be the correct credentials here.

Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

I see the following logs in Grafana:

logger=authn.password t=2024-04-19T22:10:51.793268812Z level=debug msg="Failed to authenticate password identity" client=grafana error="[password-auth.invalid] invalid password"
logger=authn.service t=2024-04-19T22:10:51.799198887Z level=info msg="Failed to authenticate request" client=auth.client.basic error="[password-auth.failed] failed to authenticate identity: [password-auth.invalid] invalid password"
logger=context userId=0 orgId=0 uname= t=2024-04-19T22:10:51.799254052Z level=info msg= error="[password-auth.failed] failed to authenticate identity: [password-auth.invalid] invalid password" remote_addr=[::1] traceID=
logger=context userId=0 orgId=0 uname= t=2024-04-19T22:10:51.799295361Z level=info msg="Request Completed" method=POST path=/api/admin/provisioning/alerting/reload status=401 remote_addr=[::1] time_ms=18 duration=18.792781ms size=121 referer= handler=/api/admin/provisioning/alerting/reload

The Alerts sidecar yields a similar error:

{"time": "2024-04-19T22:10:51.778147+00:00", "taskName": null, "msg": "Writing /etc/grafana/provisioning/alerting/alerts.yaml (ascii)", "level": "INFO"}
{"time": "2024-04-19T22:10:51.799953+00:00", "taskName": null, "msg": "POST request sent to http://localhost:3000/api/admin/provisioning/alerting/reload. Response: 401 Unauthorized {\"extra\":null,\"message\":\"Invalid username or password\",\"messageId\":\"password-auth.failed\",\"statusCode\":401,\"traceID\":\"\"}\n", "level": "DEBUG"}

Did you follow any online instructions? If so, what is the URL?

I checked Grafana auth docs.

You have persistence enabled, so old/previous admin password is there. Disable persistence, so you will loose dashboard, configs, passwords, so password from that env variable will be applied. You may try to enable persistence later.