Grafana integration with Keycloak Login failed Failed to get token from provider

Grafana Authentication
1

I have problems on Grafana authentication with Keycloak Server.

On pod “kube-prometheus-stack-grafana-0” the message error is displayed:

logger=context userId=0 orgId=0 uname= t=2025-04-01T16:09:45.10508956Z level=info msg=“Request Completed” method=GET path=/ status=302 remote_addr=10.42.2.95 time_ms=0 duration=199.627µs size=29 referer= handler=/ status_source=server
logger=context userId=0 orgId=0 uname= t=2025-04-01T16:09:46.869876946Z level=info msg=“Request Completed” method=GET path=/login/generic_oauth status=302 remote_addr=10.42.2.95 time_ms=0 duration=208.026µs size=333 referer=https://grafana-xxxxxxxxl.br/login handler=/login/:name status_source=server
logger=authn.service t=2025-04-01T16:10:12.666282174Z level=error msg=“Failed to authenticate request” client=auth.client.generic_oauth error=“[auth.oauth.token.exchange] failed to exchange code to token: Post "https://keycloak-xxxxxxxxx.br/realms/INFRA/protocol/openid-connect/token\”: dial tcp 10.103.3.15:443: i/o timeout"
logger=context userId=0 orgId=0 uname= t=2025-04-01T16:10:12.68344354Z level=info msg=“Request Completed” method=GET path=/login/generic_oauth status=302 remote_addr=10.42.2.95 time_ms=20019 duration=20.019842415s size=29 referer= handler=/login/:name status_source=server

kube-prometheus-stack helm chart version: 69.8.2
grafana version: v11.5.2
keycloak version: 26.1.4

values.yaml helm chart:

envFromSecrets:
- name: grafana-client-secret

grafana.ini:
server:
root_url: https://grafana-xxxxxxxx.br
serve_from_sub_path: false

auth.generic_oauth:
  enabled: true
  disable_login_form: true
  disable_signout_menu: false
  name: Keycloak
  allow_sign_up: true
  auto_login: false
  scopes: "email offline_access profile roles"
  groups_attribute_path: groups
  auth_url: https://keycloak.xxxxxxxx.br/realms/INFRA/protocol/openid-connect/auth
  token_url: https://keycloak.xxxxxxxx.br/realms/INFRA/protocol/openid-connect/token
  api_url: https://keycloak.xxxxxxxx.br/realms/INFRA/protocol/openid-connect/userinfo
  signout_redirect_url: https://keycloak.xxxxxxxx.br/auth/realms/INFRA/protocol/openid-connect/logout?post_logout_redirect_uri=https%3A%2F%2Fgrafana-plus.ateh.local.br%2Flogin
  role_attribute_path: contains(roles[*], 'admin') && 'Admin' || contains(roles[*], 'editor') && 'Editor' || 'Viewer',
  # role_attribute_path: "contains(realm_access.roles[*], 'grafana-admin') && 'Admin' || contains(realm_access.roles[*], 'grafana-editor') && 'Editor' || 'Viewer'"
  tls_skip_verify_insecure: true
  # allow_assign_grafana_admin: true
  # role_attribute_strict: false
  # skip_org_role_sync: false
  # use_pkce: false
  # use_refresh_token: false

What’s is happening?

2

Your Grafana pod can’t reach your Keycloak domain/url.

3

Thank You!

I have another problem now…

logger=provisioning.dashboard t=2025-04-03T11:14:00.118855399Z level=info msg="starting to provision dashboards"
logger=provisioning.dashboard t=2025-04-03T11:14:00.208396684Z level=info msg="finished to provision dashboards"
logger=provisioning.dashboard t=2025-04-03T11:14:00.224792422Z level=info msg="starting to provision dashboards"
logger=provisioning.dashboard t=2025-04-03T11:14:00.272581796Z level=info msg="finished to provision dashboards"
logger=provisioning.dashboard t=2025-04-03T11:14:00.330513372Z level=info msg="starting to provision dashboards"
logger=provisioning.dashboard t=2025-04-03T11:14:00.405530733Z level=info msg="finished to provision dashboards"
logger=authn.service t=2025-04-03T11:14:08.755125768Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2025-04-03T11:14:08.755259643Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=10.42.2.95 time_ms=1 duration=1.066036ms size=29 referer= handler=/ status_source=server
logger=authn.service t=2025-04-03T11:14:08.800163469Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=authn.service t=2025-04-03T11:14:10.239193279Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2025-04-03T11:14:10.239492881Z level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=10.42.2.95 time_ms=1 duration=1.213395ms size=434 referer=https://grafanaxxxxxxxxxxx.br/login handler=/login/:name status_source=server
logger=authn.service t=2025-04-03T11:14:10.355473431Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=authn.service t=2025-04-03T11:14:10.391864453Z level=error msg="Failed to authenticate request" client=auth.client.generic_oauth error="[auth.oauth.token.exchange] failed to exchange code to token: oauth2: \"unauthorized_client\" \"Invalid client or Invalid client credentials\""
logger=context userId=0 orgId=0 uname= t=2025-04-03T11:14:10.411335364Z level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=10.42.2.95 time_ms=56 duration=56.772513ms size=29 referer= handler=/login/:name status_source=server
logger=authn.service t=2025-04-03T11:14:10.462678864Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=infra.usagestats t=2025-04-03T11:14:56.179140781Z level=info msg="Usage stats are ready to report"
logger=provisioning.dashboard t=2025-04-03T11:14:59.873957811Z level=info msg="starting to provision dashboard

When I enable the “Client authentication” option in Keycloak for the “grafana-oauth” client, it causes the error “user token not found” and “failed to exchange code to token: oauth2: "unauthorized_client" "Invalid client or Invalid client credentials"”. The client_secret and client_id are stored in a secret in k8s with “base64” format as below:

apiVersion: v1
data:
  GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: xxxxxxxxxxxxxxxxxx
  GF_AUTH_GENERIC_OAUTH_CLIENT_ID: xxxxxxxxxxxxxxxxxxxx
kind: Secret
metadata:
  name: grafana-client-secret
  namespace: monitoring

When I disable the “Client authentication” option in keycloak, I can log in to Grafana but the admin profile is not enabled.

This is my values.yaml from kube-prometheus-stack:

grafana:
  envFromSecrets: 
    - name: grafana-client-secret
    
  grafana.ini:
    server:
      root_url: https://grafana.xxxxxxxxxxx.br
      serve_from_sub_path: false

    security:
      allow_embedding: true
    
    log:
        filters: "oauth.generic_oauth:debug"

    auth.generic_oauth:
      name: Keycloak
      enabled: true
      disable_login_form: true
      disable_signout_menu: false
      allow_sign_up: true
      auto_login: false
      client_id: grafana-oauth
      scopes: ["openid", "profile", "email", "offline_access", "roles"]
      email_attribute_path: "xxxxxx@xxxx.xxx.br"
      login_attribute_path: "plus"
      auth_url: https://keycloak.xxxxxxxxxxx.br/realms/INFRA/protocol/openid-connect/auth
      token_url: http://keycloak.keycloak/realms/INFRA/protocol/openid-connect/token
      api_url: http://keycloak.keycloak/realms/INFRA/protocol/openid-connect/userinfo
      signout_redirect_url: https://keycloak.xxxxxxxxxxx.br/auth/realms/INFRA/protocol/openid-connect/logout?post_logout_redirect_uri=https%3A%2F%2Fgrafana.xxxxxxxxxxx.br%2Flogin
      role_attribute_path: contains(roles[*], 'grafanaadmin') && 'GrafanaAdmin' || contains(roles[*], 'admin') && 'Admin' || contains(roles[*], 'editor') && 'Editor' || 'Viewer'
      allow_assign_grafana_admin: true
      role_attribute_strict: false      
      tls_skip_verify_insecure: true
      use_refresh_token: false
      use_pkce: true