Grafana integration with Keycloak - Failed to get token from provider

Grafana Configuration
1
  • What Grafana version and what operating system are you using?

Grafana v10.3.1 on Kubernetes
Keycloak 22.0.4 on Kubernetes

  • What are you trying to achieve?

I am trying to integrate Grafana with Keycloak and authenticate to Grafana via Keycloak.

  • How are you trying to achieve it?

I follwed the official Grafana documentation Grafana docs and configured Keycloak and Grafana accordingly, but when I try to login to Grafana dashboard via Keycloak, I get an error “Failed to get token from provider”

grafana error
grafana error961×1311 98.9 KB

  • What did you expect to happen?

This is my grafana.ini configuration

grafana.ini:
    server:
      domain: grafana.domain.com
      root_url: https://grafana.domain.com
    auth.generic_oauth:
      enabled: true
      name: Keycloak-SSO
      allow_sign_up: true
      allow_assign_grafana_admin: true
      use_refresh_token: true
      client_id: "client id"
      client_secret: "secret id"
      scopes: openid profile email
      email_attribute_path: email
      login_attribute_path: username
      name_attribute_path: full_name
      auth_url: https://keycloak.domain.com/auth/realms/grafana/protocol/openid-connect/auth
      token_url: https://keycloak.domain.com/auth/realms/grafana/protocol/openid-connect/token
      api_url: https://keycloak.domain.com/auth/realms/grafana/protocol/openid-connect/userinfo
      role_attribute_path: contains(roles[*], 'admin') && 'Admin' || contains(roles[*], 'editor') && 'Editor' || 'Viewer'
    auth:
      disable_login_form: false
      oauth_auto_login: false
      oauth_skip_org_role_update_sync: false
      skip_org_role_sync: false
    users:
      auto_assign_org: true
      auto_assign_org_role: Viewer  # default org_role
      auto_assign_org_id: 1
    security:
      disable_initial_admin_creation: true
      allow_embedding: true
      cookie_secure: false
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

This is a log from Grafana pod

logger=context userId=0 orgId=0 uname= t=2024-02-24T19:48:34.431573646Z level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr="IP address" time_ms=0 duration=125.679µs size=325 referer=https://grafana.domain.com/login handler=/login/:name
logger=authn.service t=2024-02-24T19:48:34.745761083Z level=error msg="Failed to authenticate request" client=auth.client.generic_oauth error="[auth.oauth.token.exchange] failed to exchange code to token: oauth2: cannot fetch token: 403 Forbidden\nResponse: <html>\r\n<head><title>403 Forbidden</title></head>\r\n<body>\r\n<center><h1>403 Forbidden</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n"
logger=context userId=0 orgId=0 uname= t=2024-02-24T19:48:34.753891725Z level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr="IP address"time_ms=61 duration=61.535069ms size=29 referer= handler=/login/:name

This is a log from Keycloak pod

2024-02-24 18:58:16,471 INFO  [org.keycloak.events] (executor-thread-94) type=CODE_TO_TOKEN, realmId=9c2779da-b509-4839-a9ae-2c8a2bfd48aa, clientId=security-admin-console, userId=b57ee5b8-1bcd-4a59-b848-960147d3e3d2, ipAddress="IP address", token_id=d9c27b25-d77f-4e6d-8416-7c31ece20e5b, grant_type=authorization_code, refresh_token_type=Refresh, scope='openid profile email', refresh_token_id=4980d486-f6e0-4b09-a650-e50a782f2dd8, code_id=7311ec97-4594-43aa-9427-ad55033998c3, client_auth_method=client-secret
2024-02-24 19:36:36,513 INFO  [org.keycloak.events] (executor-thread-92) type=LOGIN, realmId=b82a9083-349b-4fcf-953c-72cd1c04a997, clientId=grafana-oauth, userId=887efdf9-90ab-44fd-b118-2cebef8e1413, ipAddress="IP address", auth_method=openid-connect, auth_type=code, response_type=code, redirect_uri=https://grafana.domain.com/login/generic_oauth, consent=no_consent_required, code_id=7f45fa94-b7d2-40e9-bf62-248670563c49, username=test, response_mode=query, authSessionParentId=7f45fa94-b7d2-40e9-bf62-248670563c49, authSessionTabId=O6DrvfhL0W8
  • Did you follow any online instructions? If so, what is the URL?

Yes, I tried to follow a similar issue, but it looks like not my case.

The interesting thing is that when I get an error described above, I see an active fresh session of my test user in Keycloak

grafana error 2
grafana error 22231×671 107 KB

Could you please help me and point me to my issue? Thanks!

1 Like
2

In addition, when I use curl usgin the same credentials, I get a new generated token successfully

curl -s -X POST https://keycloak.domain.com/auth/realms/grafana/protocol/openid-connect/token \
 -d "grant_type=password" \
 -d "scope=openid" \
 -d "client_id=client id" \
 -d "client_secret=secret id" \
 -d "username=test" \
 -d "password=password" \
 "https://keycloak.domain.com/auth/realms/grafana/protocol/openid-connect/token" | jq .
1 Like
3

That 403 response is generated by Nginx, so find why some nginx is blocking that. Guess: there is nginx ingress, which needs some auth for inter cluster communication :person_shrugging:

4

Hi,
You were correct. We use whitelisting, when we added an exteral IP of our router to our ingress, it worked out. Thank you!

1 Like
5

im also facing this issue now,configured in gcp kubernetes cluster,can you please explain it briefly where it is blocked

1 Like
6

Hi,
I’m also facing the same issue; is it possible to provide a brief explanation?

Thank you