Grafana with Keycloak, Error: Login failed Failed to get token from provider

Grafana
21

What is a point of using oauth2-proxy in front of Grafana, which has own OIDC auth? That’s double authentication on different levels, which maybe clashing.

What is a point of using grant code flow with all these credentials issues, when you can have future proof, more secure PKCE flow?

22

Good morning @jangaraj

Thanks for your help. Sorry, I didn’t explain well. I have oauth2-proxy, to authenticate another service, like longhourn ui, for example. With Grafana I don’t using it.

I will try to implement the PKCE, but honestly I don’t understand how to do it.

Thanks again.

23

More tests I have tried this curl command:

curl -d "grant_type=password" -d "scope=openid" -d "client_id=grafana-oauth" -d "client_secret=<secret_uncoded>" -d "username=grafanadmin" -d "password=<password>" https://sso.home.domain.com/realms/home/protocol/openid-connect/token | jq .

And it seems it works fine:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4101  100  3957  100   144  20091    731 --:--:-- --:--:-- --:--:-- 21030
{
  "access_token": "<redacted>",
  "expires_in": 300,
  "refresh_expires_in": 0,
  "refresh_token": "<redacted>",
  "token_type": "Bearer",
  "id_token": "<redacted>",
  "not-before-policy": 1713371391,
  "session_state": "e741ed17-1c0e-4478-a5d4-279ecabc5757",
  "scope": "openid offline_access email profile"
}
24

FIXED! Finally! I found it checking the env variables inside the grafana pod:

kube-prometheus-stack-grafana-8f748478-5z7jr:/usr/share/grafana$ env | grep GF_
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=<secret_uncoded>
GF_PATHS_HOME=/usr/share/grafana
GF_PATHS_PROVISIONING=/etc/grafana/provisioning
GF_SECURITY_ADMIN_PASSWORD=prom-operator
GF_SECURITY_ADMIN_USER=admin
GF_PATHS_DATA=/var/lib/grafana/
GF_PATHS_LOGS=/var/log/grafana
GF_PATHS_PLUGINS=/var/lib/grafana/plugins
GF_PATHS_CONFIG=/etc/grafana/grafana.ini

Then I realized there was one miss: GF_AUTH_GENERIC_OAUTH_CLIENT_ID

And that explains why always had userId="null". So I modified the secret:

apiVersion: v1
data:
  GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: <clientsecret_base64>
  GF_AUTH_GENERIC_OAUTH_CLIENT_ID: <clientid_base64>
kind: Secret
metadata:
  creationTimestamp: null
  name: grafana-env-secret
  namespace: monitoring

And restarted the pod: kubectl -n monitoring rollout restart deployment kube-prometheus-stack-grafana

And finally, checked the envs:

kube-prometheus-stack-grafana-8f748478-5z7jr:/usr/share/grafana$ env | grep GF_
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=<secret_uncoded>
**GF_AUTH_GENERIC_OAUTH_CLIENT_ID**=grafana-oauth
GF_PATHS_HOME=/usr/share/grafana
GF_PATHS_PROVISIONING=/etc/grafana/provisioning
GF_SECURITY_ADMIN_PASSWORD=prom-operator
GF_SECURITY_ADMIN_USER=admin
GF_PATHS_DATA=/var/lib/grafana/
GF_PATHS_LOGS=/var/log/grafana
GF_PATHS_PLUGINS=/var/lib/grafana/plugins
GF_PATHS_CONFIG=/etc/grafana/grafana.ini

And boom! working:

logger=context userId=0 orgId=0 uname= t=2024-04-19T12:40:28.158783866Z level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=10.244.0.0 time_ms=0 duration=239.164µs size=345 referer= handler=/login/:name status_source=server
logger=oauth.generic_oauth t=2024-04-19T12:40:28.259028574Z level=debug msg="Getting user info"
logger=oauth.generic_oauth t=2024-04-19T12:40:28.259125406Z level=debug msg="Extracting user info from OAuth token"
logger=oauth.generic_oauth t=2024-04-19T12:40:28.259833854Z level=debug msg="Received id_token" raw_json="{\"exp\":1713530731,\"iat\":1713530431,\"auth_time\":1713529886,\"jti\":\"e74d1c01-a1ac-4fd8-ac49-0362be7c2021\",\"iss\":\"https://sso.home.domain.com/realms/home\",\"aud\":\"grafana-oauth\",\"sub\":\"d67a175d-d4ae-4d62-9070-09aa309b2ee0\",\"typ\":\"ID\",\"azp\":\"grafana-oauth\",\"session_state\":\"8dfd088e-7055-4c82-aa87-e1bc06868814\",\"at_hash\":\"mcxdogCSuT-NeDvnEoeExw\",\"acr\":\"0\",\"sid\":\"8dfd088e-7055-4c82-aa87-e1bc06868814\",\"email_verified\":true,\"roles\":[\"editor\",\"viewer\",\"offline_access\",\"admin\",\"default-roles-home.domain.com\",\"uma_authorization\"],\"name\":\"Grafana Admin\",\"preferred_username\":\"grafanadmin\",\"given_name\":\"Grafana\",\"family_name\":\"Admin\",\"email\":\"grafana@domain.com\"}" data="Name: Grafana Admin, Displayname: , Login: , Username: , Email: grafana@domain.com, Upn: , Attributes: map[]"

A happy ending!!! I hope it helps somebody else

25

In my particular case, the problem was my Grafana server not trusting the cert being returned by the Keycloak server. Resolution:

  1. Create a ConfigMap with the signing cert of the Keycloak server.
  2. Mount the ConfigMap via "extraConfigmapMounts:
  3. Reference the mounted path in “tls_client_ca” field of grafana-ini.yaml