Grafana with Keycloak- Login failed Failed to get token from provider

  • What Grafana version and what operating system are you using?
    grafana/11.3.0
  • What are you trying to achieve?
    Need to integrate grafana with keycloak with path based routing.

Grafana: https://domain.com/grafana/
Keycloak: https://domain.com/auth/

  • How are you trying to achieve it?
    using reverse proxy of application gateway i am trying to achieve this using X-forwarded option in keycloak
  • What happened?
    Login failed

Failed to get token from provider

this is my environment variables

GF_DATABASE_SSL_MODE=require
GF_SECURITY_COOKIE_SECURE=true
GF_DATABASE_TYPE=postgres
GF_PATHS_LOGS=/var/log/grafana
CONTAINER_APP_REPLICA_NAME=demografana–oo0uxcm-5544c76c5-w9884
GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
CONTAINER_APP_REVISION=demografana–oo0uxcm
GF_PATHS_PROVISIONING=/etc/grafana/provisioning
GF_PATHS_PLUGINS=/var/lib/grafana/plugins
GF_SECURITY_CSRF_TRUSTED_ORIGINS=xxxx.com
KUBERNETES_PORT_443_TCP_ADDR=100.100.224.1
GF_AUTH_GENERIC_OAUTH_API_URL=https://.com/keycloak/realms/Convatec/protocol/openid-connect/userinfo
PATH=/usr/share/grafana/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
GF_SERVER_SERVE_FROM_SUB_PATH=true
GF_SERVER_ROOT_URL=https://
.com/grafana/
GF_PATHS_CONFIG=/etc/grafana/grafana.ini
GF_PATHS_DATA=/var/lib/grafana
GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://.com/keycloak/realms/Convatec/protocol/openid-connect/auth
CONTAINER_APP_PORT=3000
KUBERNETES_PORT_443_TCP=tcp://100.100.224.1:443
KUBERNETES_SERVICE_PORT_HTTPS=443
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=*****
GF_AUTH_GENERIC_OAUTH_TOKEN_URL=https://
.com/keycloak/realms/Convatec/protocol/openid-connect/token
CONTAINER_APP_NAME=demografana

below is my grafana.ini

[server]

domain = $__env{GF_SERVER_DOMAIN}
root_url = $__env{GF_SERVER_ROOT_URL}
serve_from_sub_path = true # ${GF_SERVER_SERVE_FROM_SUB_PATH}
[database]
type = $__env{GF_DATABASE_TYPE}
host = $__env{GF_DATABASE_HOST}
name = $__env{GF_DATABASE_NAME}
user = $__env{GF_DATABASE_USER}
password = $__env{GF_DATABASE_PASSWORD}
ssl_mode = $__env{GF_DATABASE_SSL_MODE}
[auth.generic_oauth]
enabled = true
name = KEYCLOAK # this will be the oidc button name
allow_sign_up = true
client_id = grafana # this is your grafana client name on keycloak
client_secret = $__env{GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET} # check the credentials section on keycloak
scopes = openid profile email groups roles # should match with keycloak client scope setting
auth_url = $__env{GF_AUTH_GENERIC_OAUTH_AUTH_URL}
token_url = $__env{GF_AUTH_GENERIC_OAUTH_TOKEN_URL}
api_url = $__env{GF_AUTH_GENERIC_OAUTH_API_URL}
signout_redirect_url = $__env{GF_AUTH_SIGNOUT_REDIRECT_URL}
role_attribute_strict = true
role_attribute_path = contains(grafana.roles, ‘grafanaAdmin’) && ‘GrafanaAdmin’ || contains(grafana.roles, ‘admin’) && ‘Admin’ || contains(grafana.roles, ‘editor’) && ‘Editor’ || ‘Viewer’
allow_assign_grafana_admin = true
email_attribute_path = email
name_attribute_path = name
login_attribute_path = preferred_username
use_refresh_token = true
[auth]
oauth_auto_login = true
[security]
disable_initial_admin_creation = true
allow_embedding = true
cookie_secure = true
[users]
auto_assign_org = true
auto_assign_org_role = Admin # default org_role
auto_assign_org_id = 1
[log]
level = debug

  • What did you expect to happen?
    when i try to login grafana https://domain.com/grafana/ it should navigate me to keycloak login page once i login with my credential it should open grafana.
  • Can you copy/paste the configuration(s) that you are having problems with?
    mentioned above
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    2025-04-15T13:52:12.6435744Z stdout F logger=authn.service t=2025-04-15T13:52:12.643407288Z level=error msg=“Failed to authenticate request” client=auth.client.generic_oauth error=“[auth.oauth.token.exchange] failed to exchange code to token: Post "https://xxxx.com/keycloak/realms/Convatec/protocol/openid-connect/token": EOF”

2025-04-15T13:52:12.6553452Z stdout F logger=context userId=0 orgId=0 uname= t=2025-04-15T13:52:12.655181287Z level=info msg=“Request Completed” method=GET path=/login/generic_oauth status=302 remote_addr=100.100.0.129 time_ms=119 duration=119.307114ms size=37 referer= handler=/login/:name status_source=server

2025-04-15T13:52:12.9196132Z stdout F logger=authn.service t=2025-04-15T13:52:12.919407938Z level=warn msg=“Failed to authenticate request” client=auth.client.session error=“user token not found”

2025-04-15T13:52:12.9217112Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.921603747Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“any(licensing:read server.stats:read)”

2025-04-15T13:52:12.9254720Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.925348953Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“action:dashboards:read scopes:”

2025-04-15T13:52:12.9254925Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.925384533Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“any(folders:read folders:create dashboards:read dashboards:create)”

2025-04-15T13:52:12.9255025Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.925394468Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“action:datasources:explore scopes:”

2025-04-15T13:52:12.9255291Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.925404856Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“any(alert.rules:read alert.rules.external:read)”

2025-04-15T13:52:12.9255337Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.92541597Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“any(alert.notifications:read alert.notifications.external:read)”

2025-04-15T13:52:12.9255713Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.925425807Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“any(alert.notifications:read alert.notifications.external:read)”

2025-04-15T13:52:12.9255761Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.92543576Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“any(alert.instances:read alert.instances.external:read alert.silences:read)”

2025-04-15T13:52:12.9255794Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.92544488Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“any(alert.instances:read alert.instances.external:read)”

2025-04-15T13:52:12.9255818Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.925455619Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“any(alert.rules:create alert.rules.external:write)”

2025-04-15T13:52:12.9255896Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.92547419Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“any(datasources:create all of datasources:read, any of datasources:delete, datasources:write)”

2025-04-15T13:52:12.9256036Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.925486646Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“any(all of orgs:read, orgs:write all of orgs.preferences:read, orgs.preferences:write)”

2025-04-15T13:52:12.9256070Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.925496626Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“action:settings:read scopes:settings:*”

2025-04-15T13:52:12.9256096Z stdout F logger=context userId=0 orgId=0 uname= t=2025-04-15T13:52:12.925507933Z level=debug msg=“Failed to authenticate user in global scope” error=“[auth.identity.unsupported] invalid identity type”

2025-04-15T13:52:12.9256145Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.92552873Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“action:settings:read scopes:settings:*”

2025-04-15T13:52:12.9256164Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.925538236Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“any(plugins:write plugins:install)”

2025-04-15T13:52:12.9256238Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.925555806Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“action:datasources:explore scopes:”

2025-04-15T13:52:12.9256353Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.925572659Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“any(org.users:read users:read)”

2025-04-15T13:52:12.9256386Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.925581538Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“any(teams:create all of teams:read, any of teams:write, teams.permissions:write, teams.permissions:read)”

2025-04-15T13:52:12.9256425Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.925587505Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“any(serviceaccounts:read serviceaccounts:create)”

2025-04-15T13:52:12.9296215Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.929487935Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“action:apikeys:read scopes:”

2025-04-15T13:52:12.9296428Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.92952571Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“any(settings:read settings:write settings:read settings:write settings:read settings:write settings:read settings:write settings:read settings:write settings:read settings:write)”

2025-04-15T13:52:12.9296516Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.929543655Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“any(support.bundles:read support.bundles:create)”

2025-04-15T13:52:12.9316760Z stdout F logger=accesscontrol t=2025-04-15T13:52:12.931564085Z level=debug msg=“No permissions set” id=:0 orgID=0 permissions=“any(dashboards:create folders:create)”

2025-04-15T13:52:20.0026907Z stdout F logger=ngalert.scheduler t=2025-04-15T13:52:20.002511708Z level=debug msg=“Alert rules fetched” rulesCount=0 foldersCount=0 updatedRules=0

2025-04-15T13:52:30.0039625Z stdout F logger=ngalert.scheduler t=2025-04-15T13:52:30.003718595Z level=debug msg=“Alert rules fetched” rulesCount=0 foldersCount=0 updatedRules=0

  • Did you follow any online instructions? If so, what is the URL?

Are you sure that your Grafana has network connectivity to https://*xxxx*.com/keycloak/realms/Convatec/protocol/openid-connect/token?

(post deleted by author)

After updated the firewall rule it fixed