Cannot log into Grafana after changing password using grafana-cli

  • What Grafana version and what operating system are you using?
    Grafana 9.3.1 installed using the Kube-Prometheus-Stack Helm chart

  • What are you trying to achieve?
    I am trying to update the admin password using the grafana-cli’s reset-admin-password command.

  • How are you trying to achieve it?
    Using the following command to invoke the grafana-cli:
    kubectl exec -n -c grafana – bin/grafana-cli admin reset-admin-password

  • What happened?
    The command runs and says that the password was updated successfully.

  • What did you expect to happen?
    I expected to be able to log into Grafana using the admin User ID and the updated password.

  • Can you copy/paste the configuration(s) that you are having problems with?

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    Steps to reproduce:

  1. Deployed Grafana using the kube-prometheus-stack Helm chart - we have a random generated password created for our users (in this case - NGE4MTFmNTFkNDJiZTlmODFkMDkwMmFk). Was able to confirm that I was able to log in with that password.
  2. Run the grafana-cli to update the password to “BrandNewPassword”:
    logger=settings t=2023-04-03T19:17:17.981925228Z level=info msg=“Starting Grafana” version= commit= branch= compiled=1970-01-01T00:00:00Z
    logger=settings t=2023-04-03T19:17:17.982302042Z level=info msg=“Config loaded from” file=/usr/share/grafana/conf/defaults.ini
    logger=settings t=2023-04-03T19:17:17.982336912Z level=info msg=“Config overridden from Environment variable” var=“GF_PATHS_DATA=/var/lib/grafana/”
    logger=settings t=2023-04-03T19:17:17.982349256Z level=info msg=“Config overridden from Environment variable” var=“GF_PATHS_LOGS=/var/log/grafana”
    logger=settings t=2023-04-03T19:17:17.982363399Z level=info msg=“Config overridden from Environment variable” var=“GF_PATHS_PLUGINS=/var/lib/grafana/plugins”
    logger=settings t=2023-04-03T19:17:17.982375415Z level=info msg=“Config overridden from Environment variable” var=“GF_PATHS_PROVISIONING=/etc/grafana/provisioning”
    logger=settings t=2023-04-03T19:17:17.982391871Z level=info msg=“Config overridden from Environment variable” var=“GF_SECURITY_ADMIN_USER=admin”
    logger=settings t=2023-04-03T19:17:17.982402696Z level=info msg=“Config overridden from Environment variable” var=“GF_SECURITY_ADMIN_PASSWORD="
    logger=settings t=2023-04-03T19:17:17.98241914Z level=info msg=“Path Home” path=/usr/share/grafana
    logger=settings t=2023-04-03T19:17:17.982429189Z level=info msg=“Path Data” path=/var/lib/grafana/
    logger=settings t=2023-04-03T19:17:17.982439378Z level=info msg=“Path Logs” path=/var/log/grafana
    logger=settings t=2023-04-03T19:17:17.982447998Z level=info msg=“Path Plugins” path=/var/lib/grafana/plugins
    logger=settings t=2023-04-03T19:17:17.982457275Z level=info msg=“Path Provisioning” path=/etc/grafana/provisioning
    logger=settings t=2023-04-03T19:17:17.98247754Z level=info msg=“App mode production”
    logger=settings t=2023-04-03T19:17:17.982490776Z level=info msg=“Config loaded from” file=/usr/share/grafana/conf/defaults.ini
    logger=settings t=2023-04-03T19:17:17.982500631Z level=info msg=“Config overridden from Environment variable” var=“GF_PATHS_DATA=/var/lib/grafana/”
    logger=settings t=2023-04-03T19:17:17.98250982Z level=info msg=“Config overridden from Environment variable” var=“GF_PATHS_LOGS=/var/log/grafana”
    logger=settings t=2023-04-03T19:17:17.982520272Z level=info msg=“Config overridden from Environment variable” var=“GF_PATHS_PLUGINS=/var/lib/grafana/plugins”
    logger=settings t=2023-04-03T19:17:17.982528799Z level=info msg=“Config overridden from Environment variable” var=“GF_PATHS_PROVISIONING=/etc/grafana/provisioning”
    logger=settings t=2023-04-03T19:17:17.982537655Z level=info msg=“Config overridden from Environment variable” var=“GF_SECURITY_ADMIN_USER=admin”
    logger=settings t=2023-04-03T19:17:17.982551362Z level=info msg=“Config overridden from Environment variable” var="GF_SECURITY_ADMIN_PASSWORD=

    logger=settings t=2023-04-03T19:17:17.982560702Z level=info msg=“Path Home” path=/usr/share/grafana
    logger=settings t=2023-04-03T19:17:17.982569735Z level=info msg=“Path Data” path=/var/lib/grafana/
    logger=settings t=2023-04-03T19:17:17.982579817Z level=info msg=“Path Logs” path=/var/log/grafana
    logger=settings t=2023-04-03T19:17:17.982588403Z level=info msg=“Path Plugins” path=/var/lib/grafana/plugins
    logger=settings t=2023-04-03T19:17:17.982596769Z level=info msg=“Path Provisioning” path=/etc/grafana/provisioning
    logger=settings t=2023-04-03T19:17:17.982606074Z level=info msg=“App mode production”
    logger=secrets t=2023-04-03T19:17:17.982964686Z level=info msg=“Envelope encryption state” enabled=true currentprovider=secretKey.v1

Admin password changed successfully :heavy_check_mark:

Tried to log into Grafana using “BrandNewPassword” - invalid user ID/password:

Tried to log into Grafana using the old password - invalid user ID/password:

I checked the Kubernetes secret and the admin user ID and password are what I expected:

	"admin-password":"QnJhbmROZXdQYXNzd29yZA==", (BrandNewPassword)

“admin-user”:“YWRtaW4=” (admin)

I checked the values in the Grafana pod and they pointed to the secret:
GF_PATHS_PLUGINS: /var/lib/grafana/plugins
GF_PATHS_PROVISIONING: /etc/grafana/provisioning

The environment variables in the Grafana pod were also correct:
kubectl exec -n monitoring v4m-grafana-79cbdfc558-46m6k – printenv
Defaulted container “grafana-sc-dashboard” out of: grafana-sc-dashboard, grafana-sc-datasources, grafana, init-chown-data (init)
PATH=/app/.venv/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=v4m-grafana-79cbdfc558-46m6k

REQ_USERNAME=admin
REQ_PASSWORD=BrandNewPassword

I’m not sure where else to work; however, this used to work when we were on Grafana 9.0.2. I understand that, if you know the password, you should use the Grafana UI; however, we want to have this option available when new administrators inherit the monitoring components in an environment.

How can we get the grafana-cli to properly update the password so that users can log in using the new password?

Update to this issue:

  • If I change the password in the Grafana UI (clicking on the user icon → Change Password and restart the pods, I get the same behavior - I’m not able to log into Grafana with either the old or the new password.

I’m also seeing entries like the one below in my Grafana pods:
kubectl logs -n monitoring v4m-grafana-796586f548-vj8gz grafana
{“error”:“too many consecutive incorrect login attempts for user - login for user temporarily blocked”,“level”:“error”,“logger”:“context”,“msg”:“invalid username or password”,“t”:“2023-04-10T18:50:51.418872149Z”,“traceID”:“”}
{“duration”:“3.907476ms”,“level”:“info”,“logger”:“context”,“method”:“POST”,“msg”:“Request Completed”,“orgId”:0,“path”:“/api/admin/provisioning/dashboards/reload”,“referer”:“”,“remote_addr”:“[::1]”,“size”:56,“status”:401,“t”:“2023-04-10T18:50:51.41901959Z”,“time_ms”:3,“uname”:“”,“userId”:0}

Has anything changed with the password change functionality between the older 9.0.2 and 9.3.1? Any help would be greatly appreciated!