I have problems on Grafana authentication with Keycloak Server.
On pod “kube-prometheus-stack-grafana-0” the message error is displayed:
logger=context userId=0 orgId=0 uname= t=2025-04-01T16:09:45.10508956Z level=info msg=“Request Completed” method=GET path=/ status=302 remote_addr=10.42.2.95 time_ms=0 duration=199.627µs size=29 referer= handler=/ status_source=server
logger=context userId=0 orgId=0 uname= t=2025-04-01T16:09:46.869876946Z level=info msg=“Request Completed” method=GET path=/login/generic_oauth status=302 remote_addr=10.42.2.95 time_ms=0 duration=208.026µs size=333 referer=https://grafana-xxxxxxxxl.br/login handler=/login/:name status_source=server
logger=authn.service t=2025-04-01T16:10:12.666282174Z level=error msg=“Failed to authenticate request” client=auth.client.generic_oauth error=“[auth.oauth.token.exchange] failed to exchange code to token: Post "https://keycloak-xxxxxxxxx.br/realms/INFRA/protocol/openid-connect/token\”: dial tcp 10.103.3.15:443: i/o timeout"
logger=context userId=0 orgId=0 uname= t=2025-04-01T16:10:12.68344354Z level=info msg=“Request Completed” method=GET path=/login/generic_oauth status=302 remote_addr=10.42.2.95 time_ms=20019 duration=20.019842415s size=29 referer= handler=/login/:name status_source=server
kube-prometheus-stack helm chart version: 69.8.2
grafana version: v11.5.2
keycloak version: 26.1.4
values.yaml helm chart:
envFromSecrets:
- name: grafana-client-secret
grafana.ini:
server:
root_url: https://grafana-xxxxxxxx.br
serve_from_sub_path: false
auth.generic_oauth:
enabled: true
disable_login_form: true
disable_signout_menu: false
name: Keycloak
allow_sign_up: true
auto_login: false
scopes: "email offline_access profile roles"
groups_attribute_path: groups
auth_url: https://keycloak.xxxxxxxx.br/realms/INFRA/protocol/openid-connect/auth
token_url: https://keycloak.xxxxxxxx.br/realms/INFRA/protocol/openid-connect/token
api_url: https://keycloak.xxxxxxxx.br/realms/INFRA/protocol/openid-connect/userinfo
signout_redirect_url: https://keycloak.xxxxxxxx.br/auth/realms/INFRA/protocol/openid-connect/logout?post_logout_redirect_uri=https%3A%2F%2Fgrafana-plus.ateh.local.br%2Flogin
role_attribute_path: contains(roles[*], 'admin') && 'Admin' || contains(roles[*], 'editor') && 'Editor' || 'Viewer',
# role_attribute_path: "contains(realm_access.roles[*], 'grafana-admin') && 'Admin' || contains(realm_access.roles[*], 'grafana-editor') && 'Editor' || 'Viewer'"
tls_skip_verify_insecure: true
# allow_assign_grafana_admin: true
# role_attribute_strict: false
# skip_org_role_sync: false
# use_pkce: false
# use_refresh_token: false
What’s is happening?
Your Grafana pod can’t reach your Keycloak domain/url.
1 Like
Thank You!
I have another problem now…
logger=provisioning.dashboard t=2025-04-03T11:14:00.118855399Z level=info msg="starting to provision dashboards"
logger=provisioning.dashboard t=2025-04-03T11:14:00.208396684Z level=info msg="finished to provision dashboards"
logger=provisioning.dashboard t=2025-04-03T11:14:00.224792422Z level=info msg="starting to provision dashboards"
logger=provisioning.dashboard t=2025-04-03T11:14:00.272581796Z level=info msg="finished to provision dashboards"
logger=provisioning.dashboard t=2025-04-03T11:14:00.330513372Z level=info msg="starting to provision dashboards"
logger=provisioning.dashboard t=2025-04-03T11:14:00.405530733Z level=info msg="finished to provision dashboards"
logger=authn.service t=2025-04-03T11:14:08.755125768Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2025-04-03T11:14:08.755259643Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=10.42.2.95 time_ms=1 duration=1.066036ms size=29 referer= handler=/ status_source=server
logger=authn.service t=2025-04-03T11:14:08.800163469Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=authn.service t=2025-04-03T11:14:10.239193279Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2025-04-03T11:14:10.239492881Z level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=10.42.2.95 time_ms=1 duration=1.213395ms size=434 referer=https://grafanaxxxxxxxxxxx.br/login handler=/login/:name status_source=server
logger=authn.service t=2025-04-03T11:14:10.355473431Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=authn.service t=2025-04-03T11:14:10.391864453Z level=error msg="Failed to authenticate request" client=auth.client.generic_oauth error="[auth.oauth.token.exchange] failed to exchange code to token: oauth2: \"unauthorized_client\" \"Invalid client or Invalid client credentials\""
logger=context userId=0 orgId=0 uname= t=2025-04-03T11:14:10.411335364Z level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=10.42.2.95 time_ms=56 duration=56.772513ms size=29 referer= handler=/login/:name status_source=server
logger=authn.service t=2025-04-03T11:14:10.462678864Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=infra.usagestats t=2025-04-03T11:14:56.179140781Z level=info msg="Usage stats are ready to report"
logger=provisioning.dashboard t=2025-04-03T11:14:59.873957811Z level=info msg="starting to provision dashboard
When I enable the “Client authentication” option in Keycloak for the “grafana-oauth” client, it causes the error “user token not found” and “failed to exchange code to token: oauth2: "unauthorized_client" "Invalid client or Invalid client credentials"”. The client_secret and client_id are stored in a secret in k8s with “base64” format as below:
apiVersion: v1
data:
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: xxxxxxxxxxxxxxxxxx
GF_AUTH_GENERIC_OAUTH_CLIENT_ID: xxxxxxxxxxxxxxxxxxxx
kind: Secret
metadata:
name: grafana-client-secret
namespace: monitoring
When I disable the “Client authentication” option in keycloak, I can log in to Grafana but the admin profile is not enabled.
This is my values.yaml from kube-prometheus-stack:
grafana:
envFromSecrets:
- name: grafana-client-secret
grafana.ini:
server:
root_url: https://grafana.xxxxxxxxxxx.br
serve_from_sub_path: false
security:
allow_embedding: true
log:
filters: "oauth.generic_oauth:debug"
auth.generic_oauth:
name: Keycloak
enabled: true
disable_login_form: true
disable_signout_menu: false
allow_sign_up: true
auto_login: false
client_id: grafana-oauth
scopes: ["openid", "profile", "email", "offline_access", "roles"]
email_attribute_path: "xxxxxx@xxxx.xxx.br"
login_attribute_path: "plus"
auth_url: https://keycloak.xxxxxxxxxxx.br/realms/INFRA/protocol/openid-connect/auth
token_url: http://keycloak.keycloak/realms/INFRA/protocol/openid-connect/token
api_url: http://keycloak.keycloak/realms/INFRA/protocol/openid-connect/userinfo
signout_redirect_url: https://keycloak.xxxxxxxxxxx.br/auth/realms/INFRA/protocol/openid-connect/logout?post_logout_redirect_uri=https%3A%2F%2Fgrafana.xxxxxxxxxxx.br%2Flogin
role_attribute_path: contains(roles[*], 'grafanaadmin') && 'GrafanaAdmin' || contains(roles[*], 'admin') && 'Admin' || contains(roles[*], 'editor') && 'Editor' || 'Viewer'
allow_assign_grafana_admin: true
role_attribute_strict: false
tls_skip_verify_insecure: true
use_refresh_token: false
use_pkce: true
Did you get anywhere with this? I’m having the exact same issue as of today. It worked fine previously, but today I’m getting the same errors as you in the logs.
For anyone who finds this via google, the solution was stupid to me, I would get invalid_code in keycloak on sign-in like this;
grant_type
authorization_code
code_id
some-long-uuid
client_auth_method
client-secret
error
invalid_code
The reason? My client id was called grafana-oidc, changing it to grafana instead and updating the container and bam, works instantly.
Grafana v12.3.1 (0d1a5b4420)
Keycloak 26.4.7 deployed via operator.
