Grafana with Keycloak, Error: Login failed Failed to get token from provider

i5js · April 18, 2024, 8:05am

Hello all,

After googling a lot, and tried a lot of “fixes” I also found, I have decided to create an account and ask for help, because I’m running out of ideas. I’m getting the error message: Error: Login failed Failed to get token from provider, as I said, I have read a lot, but none of fixes I found worked in my environment. It’s a Kubernetes cluster, v1.27.4. I have deployed Keycloak using helm, and grafana as part of kube-prometheus-stack-58.1.1, using helm too. Below you can find the technical details:

Prometheus values, regarding grafana.ini.

grafana:
  # Configuring grafana to run behind a proxy under a subpath
  envFromSecret: grafana-env-secret
  # Additional data source
  additionalDataSources:
  - name: Loki
    type: loki
    url: http://loki-gateway.logging.svc.cluster.local
  grafana.ini:
      server:
        # Configuring /grafana subpath
        domain: monitoring.home.domain.com
        root_url: "https://%(domain)s/grafana/"
        # rewrite rules configured in nginx rules
        # https://grafana.com/tutorials/run-grafana-behind-a-proxy/
        serve_from_sub_path: false
      log:
        filters: "oauth.generic_oauth:debug"
      # SSO configuration
      auth.generic_oauth:
        enabled: true
        name: Keycloak-OAuth
        allow_sign_up: true
        client_id: grafana-oauth
        scopes: openid email profile offline_access roles
        email_attribute_path: email
        login_attribute_path: username
        name_attribute_path: full_name
        auth_url: https://sso.home.domain.com/realms/home/protocol/openid-connect/auth
        token_url: https://sso.home.domain.com/realms/home/protocol/openid-connect/token
        api_url: https://sso.home.domain.com/realms/home/protocol/openid-connect/userinfo
        role_attribute_path: contains(roles[*], 'grafanaadmin') && 'GrafanaAdmin' || contains(roles[*], 'admin') && 'Admin' || contains(roles[*], 'editor') && 'Editor' || 'Viewer'
        signout_redirect_url: https://sso.home.domain.com/realms/home/protocol/openid-connect/logout?client_id=grafana&post_logout_redirect_uri=https%3A%2F%2Fmonitoring.home.domain.com%2Fgrafana%2Flogin%2Fgeneric_oauth

Client config in Keycloak

Logs from Grafana Pod

logger=authn.service t=2024-04-18T07:00:22.267756191Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=authn.service t=2024-04-18T07:00:25.975811382Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2024-04-18T07:00:25.976396749Z level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=10.244.0.0 time_ms=3 duration=3.291122ms size=345 referer=https://monitoring.home.javisalas.com/grafana/login handler=/login/:name status_source=server
logger=authn.service t=2024-04-18T07:00:37.716336874Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=authn.service t=2024-04-18T07:00:37.781162692Z level=error msg="Failed to authenticate request" client=auth.client.generic_oauth error="[auth.oauth.token.exchange] failed to exchange code to token: oauth2: \"unauthorized_client\" \"Invalid client or Invalid client credentials\""

Logs from Keycloak pod:

024-04-17 18:23:25,095 WARN  [org.keycloak.events] (executor-thread-149) type="CODE_TO_TOKEN_ERROR", realmId="2bb4892c-4381-470d-bc6d-5aa45108067f", clientId="grafana-oauth", userId="null", ipAddress="10.244.5.0", error="invalid_client_credentials", grant_type="authorization_code"
2024-04-17 18:23:25,099 WARN  [org.keycloak.events] (executor-thread-149) type="CODE_TO_TOKEN_ERROR", realmId="2bb4892c-4381-470d-bc6d-5aa45108067f", clientId="grafana-oauth", userId="null", ipAddress="10.244.5.0", error="invalid_client_credentials", grant_type="authorization_code"
2024-04-18 07:00:40,002 WARN  [org.keycloak.events] (executor-thread-151) type="CODE_TO_TOKEN_ERROR", realmId="2bb4892c-4381-470d-bc6d-5aa45108067f", clientId="grafana-oauth", userId="null", ipAddress="10.244.5.0", error="invalid_client_credentials", grant_type="authorization_code"
2024-04-18 07:00:40,008 WARN  [org.keycloak.events] (executor-thread-151) type="CODE_TO_TOKEN_ERROR", realmId="2bb4892c-4381-470d-bc6d-5aa45108067f", clientId="grafana-oauth", userId="null", ipAddress="10.244.5.0", error="invalid_client_credentials", grant_type="authorization_code"

Logs from Nginx reverse proxy pod


10.244.0.0 -  [18/Apr/2024:07:00:40 +0000] "GET /grafana/login HTTP/2.0" 200 37794 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 163 0.091 [monitoring-kube-prometheus-stack-grafana-80] [] 10.244.5.230:3000 37848 0.091 200 80d4c714b41e3d36dd3f54d01cba1e0e
10.244.0.0 -  [18/Apr/2024:07:00:40 +0000] "GET /grafana/public/img/grafana_icon.svg HTTP/2.0" 304 0 "https://monitoring.home.domain.com/grafana/login" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 67 0.001 [monitoring-kube-prometheus-stack-grafana-80] [] 10.244.5.230:3000 0 0.001 304 37a3c196d7f3c466b2c7d3daf9d9e3e6
10.244.0.0 -  [18/Apr/2024:07:00:40 +0000] "GET /grafana/public/fonts/inter/Inter-Regular.woff2 HTTP/2.0" 304 0 "https://monitoring.home.domain.com/grafana/login" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 84 0.002 [monitoring-kube-prometheus-stack-grafana-80] [] 10.244.5.230:3000 0 0.002 304 f8d601c1dac31784ce5232ec51b47052
10.244.0.0 -  [18/Apr/2024:07:00:40 +0000] "GET /grafana/public/fonts/inter/Inter-Medium.woff2 HTTP/2.0" 304 0 "https://monitoring.home.domain.com/grafana/login" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 74 0.002 [monitoring-kube-prometheus-stack-grafana-80] [] 10.244.5.230:3000 0 0.001 304 028d166ad2f9de56e4de7820efe1bc85
10.244.0.0 -  [18/Apr/2024:07:01:16 +0000] "GET /admin/master/console/ HTTP/2.0" 200 2748 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 29 0.001 [keycloak-keycloak-http] [] 10.244.4.210:8080 2748 0.001 200 82914359b16629849f910faa6e2dca9f
10.244.0.0 -  [18/Apr/2024:07:01:16 +0000] "GET /realms/master/protocol/openid-connect/3p-cookies/step1.html HTTP/2.0" 200 2318 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 61 0.001 [keycloak-keycloak-http] [] 10.244.4.210:8080 2318 0.001 200 3e467e8c022553018687b4a0dc435d2a
10.244.0.0 -  [18/Apr/2024:07:01:16 +0000] "GET /realms/master/protocol/openid-connect/3p-cookies/step2.html HTTP/2.0" 200 686 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 112 0.000 [keycloak-keycloak-http] [] 10.244.4.210:8080 686 0.001 200 b1583b59cfb0acd83c94530e7d5508ac
10.244.0.0 -  [18/Apr/2024:07:01:16 +0000] "GET /realms/master/protocol/openid-connect/login-status-iframe.html HTTP/2.0" 200 4006 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 57 0.001 [keycloak-keycloak-http] [] 10.244.4.210:8080 4006 0.001 200 4e7912a8679b120f850f0d2950e8f28b
10.244.0.0 -  [18/Apr/2024:07:01:16 +0000] "GET /realms/master/protocol/openid-connect/login-status-iframe.html/init?client_id=security-admin-console&origin=https%3A%2F%2Fsso.home.domain.com HTTP/2.0" 204 0 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 126 0.001 [keycloak-keycloak-http] [] 10.244.4.210:8080 0 0.000 204 73cf2bbec26bdad299cc2a6357c1ac5b
10.244.0.0 -  [18/Apr/2024:07:01:16 +0000] "GET /realms/master/protocol/openid-connect/auth?client_id=security-admin-console&redirect_uri=https%3A%2F%2Fsso.home.domain.com%2Fadmin%2Fmaster%2Fconsole%2F%23%2Fhome%2Fclients%2Fd14fa3e0-0c1f-43a5-8358-e95799ac0e99%2Fsessions&state=d02b6b3e-4457-437b-885c-2c240c1e30c3&response_mode=fragment&response_type=code&scope=openid&nonce=e6f19f74-ce34-48e3-8563-d0166b146f98&code_challenge=_3ja1treMqNVtuYZQybxIcweC7__tSC3OYF5azil2uw&code_challenge_method=S256 HTTP/2.0" 200 4564 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 338 0.002 [keycloak-keycloak-http] [] 10.244.4.210:8080 4564 0.002 200 184b84ba939b1ad816cf9f3d9c9907f9
10.244.0.0 -  [18/Apr/2024:07:01:22 +0000] "POST /realms/master/login-actions/authenticate?session_code=L1hU6QmpauWO6fN-pLFGkhWGO4iRCluPFlRXOCwOiZg&execution=a13783f1-f7a6-44b5-aa71-221971add10c&client_id=security-admin-console&tab_id=_fSGWzhxLA0 HTTP/2.0" 200 5115 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 1302 0.025 [keycloak-keycloak-http] [] 10.244.4.210:8080 5115 0.025 200 19f904daa8a17c0aa9b7fe8619a83258
10.244.0.0 -  [18/Apr/2024:07:01:32 +0000] "POST /realms/master/login-actions/authenticate?session_code=zwsoGXkWwVGZ6Cg4b5PvYRcohyzhBMUHWaOP_2vxs88&execution=a13783f1-f7a6-44b5-aa71-221971add10c&client_id=security-admin-console&tab_id=_fSGWzhxLA0 HTTP/2.0" 302 0 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 407 0.026 [keycloak-keycloak-http] [] 10.244.4.210:8080 0 0.025 302 5f5d14a070e9130e519734775971c197
10.244.0.0 -  [18/Apr/2024:07:01:32 +0000] "GET /admin/master/console/ HTTP/2.0" 200 2748 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 29 0.001 [keycloak-keycloak-http] [] 10.244.4.210:8080 2748 0.001 200 d4fbea86ec2b6e2907624cc44953f7a2
10.244.0.0 -  [18/Apr/2024:07:01:32 +0000] "GET /realms/master/protocol/openid-connect/3p-cookies/step1.html HTTP/2.0" 200 2318 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 1275 0.001 [keycloak-keycloak-http] [] 10.244.4.210:8080 2318 0.000 200 16e797ce8e94917dd765b9de1efac248
10.244.0.0 -  [18/Apr/2024:07:01:32 +0000] "GET /realms/master/protocol/openid-connect/3p-cookies/step2.html HTTP/2.0" 200 686 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 80 0.001 [keycloak-keycloak-http] [] 10.244.4.210:8080 686 0.001 200 bad65e518eb242d92a300d1de38b2543
10.244.0.0 -  [18/Apr/2024:07:01:32 +0000] "GET /realms/master/protocol/openid-connect/login-status-iframe.html HTTP/2.0" 200 4006 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 63 0.000 [keycloak-keycloak-http] [] 10.244.4.210:8080 4006 0.000 200 3238046283027c2d98e4fe0d70c6a994
10.244.0.0 -  [18/Apr/2024:07:01:32 +0000] "POST /realms/master/protocol/openid-connect/token HTTP/2.0" 200 3303 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 564 0.005 [keycloak-keycloak-http] [] 10.244.4.210:8080 3303 0.005 200 c1e4b37f95fd535d018f5c5555fc3ba9
10.244.0.0 -  [18/Apr/2024:07:01:32 +0000] "GET /resources/master/admin/en HTTP/2.0" 200 276212 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 42 0.014 [keycloak-keycloak-http] [] 10.244.4.210:8080 276489 0.014 200 daebd6096bffd08ac3aac1b54b4800b1
10.244.0.0 -  [18/Apr/2024:07:01:32 +0000] "GET /realms/master/protocol/openid-connect/login-status-iframe.html/init?client_id=security-admin-console&origin=https%3A%2F%2Fsso.home.domain.com HTTP/2.0" 204 0 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 231 0.000 [keycloak-keycloak-http] [] 10.244.4.210:8080 0 0.000 204 14f4115319a4cc2485271fff77db82ad
10.244.0.0 -  [18/Apr/2024:07:01:32 +0000] "GET /admin/master/console/whoami?currentRealm=home HTTP/2.0" 200 456 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 1040 0.001 [keycloak-keycloak-http] [] 10.244.4.210:8080 456 0.001 200 1b680cb8df08fee2c6c3db0b2e3e0024
10.244.0.0 -  [18/Apr/2024:07:01:32 +0000] "GET /admin/serverinfo HTTP/2.0" 200 230354 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 981 0.008 [keycloak-keycloak-http] [] 10.244.4.210:8080 230590 0.008 200 588a88a71164e95efd1b468f38aa5ab7
10.244.0.0 -  [18/Apr/2024:07:01:32 +0000] "GET /admin/realms/home/clients/d14fa3e0-0c1f-43a5-8358-e95799ac0e99 HTTP/2.0" 200 2098 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 1014 0.002 [keycloak-keycloak-http] [] 10.244.4.210:8080 2098 0.002 200 4bf71dad3989296d9f9f446aa855e9cb
10.244.0.0 -  [18/Apr/2024:07:01:32 +0000] "GET /admin/realms/home/clients/d14fa3e0-0c1f-43a5-8358-e95799ac0e99/user-sessions?first=0&max=11 HTTP/2.0" 200 280 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 1034 0.002 [keycloak-keycloak-http] [] 10.244.4.210:8080 280 0.002 200 9a91b14c0cdd078c5720a49ef6e9d641
10.244.0.0 -  [18/Apr/2024:07:01:32 +0000] "GET /admin/realms/home/clients/d14fa3e0-0c1f-43a5-8358-e95799ac0e99/offline-sessions?first=0&max=11 HTTP/2.0" 200 2 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 1036 0.003 [keycloak-keycloak-http] [] 10.244.4.210:8080 2 0.003 200 eb2b20a4c2410dbc330e098afafbce6a
10.244.0.0 -  [18/Apr/2024:07:01:36 +0000] "GET /admin/realms/home/users/profile HTTP/2.0" 200 878 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 992 0.001 [keycloak-keycloak-http] [] 10.244.4.210:8080 878 0.002 200 f8aa92864e612191dc3754f459cc786c
10.244.0.0 -  [18/Apr/2024:07:01:36 +0000] "GET /admin/realms/home/components?type=org.keycloak.storage.UserStorageProvider HTTP/2.0" 200 2 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 1023 0.001 [keycloak-keycloak-http] [] 10.244.4.210:8080 2 0.002 200 d3465df833a68d9e9e1d5d85e6770791
10.244.0.0 -  [18/Apr/2024:07:01:36 +0000] "GET /admin/realms/home HTTP/2.0" 200 4434 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 982 0.005 [keycloak-keycloak-http] [] 10.244.4.210:8080 4434 0.006 200 052a7eec161a104218ea013d8b4608c0
10.244.0.0 -  [18/Apr/2024:07:01:36 +0000] "GET /admin/realms/home/ui-ext/brute-force-user?briefRepresentation=true&first=0&max=11&q= HTTP/2.0" 200 807 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 1029 0.003 [keycloak-keycloak-http] [] 10.244.4.210:8080 807 0.004 200 89ab203fe4598df225a9469bd523b99f
10.244.0.0 -  [18/Apr/2024:07:01:39 +0000] "GET /admin/realms/home HTTP/2.0" 200 4434 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 982 0.001 [keycloak-keycloak-http] [] 10.244.4.210:8080 4434 0.001 200 64578af83bd1417a658cbc39afc07ad6
10.244.0.0 -  [18/Apr/2024:07:01:39 +0000] "GET /admin/realms/home/users/d67a175d-d4ae-4d62-9070-09aa309b2ee0?userProfileMetadata=true HTTP/2.0" 200 669 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 1030 0.002 [keycloak-keycloak-http] [] 10.244.4.210:8080 669 0.002 200 bceca47e93279231d7bdb6bf4cfddf16
10.244.0.0 -  [18/Apr/2024:07:01:39 +0000] "GET /admin/realms/home/attack-detection/brute-force/users/d67a175d-d4ae-4d62-9070-09aa309b2ee0 HTTP/2.0" 200 72 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 1032 0.002 [keycloak-keycloak-http] [] 10.244.4.210:8080 72 0.002 200 4a5fdc8e312794741f4d5aea99fdea38
10.244.0.0 -  [18/Apr/2024:07:01:39 +0000] "GET /admin/realms/home/users/profile HTTP/2.0" 200 878 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 992 0.001 [keycloak-keycloak-http] [] 10.244.4.210:8080 878 0.001 200 a9b039fd8be559f50996a17c82d80008
10.244.0.0 -  [18/Apr/2024:07:01:39 +0000] "GET /admin/realms/home/authentication/required-actions HTTP/2.0" 200 1384 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15" 1004 0.002 [keycloak-keycloak-http] [] 10.244.4.210:8080 1384 0.002 200 a3a074436061cf22f4621d64feac5135

Anybody can please help me? I’m completly lost now.

Many thanks in advanced,

jangaraj · April 18, 2024, 12:45pm

You don’t have client secret in your config, so invalid_client_credentials is correct.

I would use public client in the Keycloak and enable pkce auth in Grafana = no client secret wiol be required.

i5js · April 18, 2024, 1:00pm

Hey @jangaraj Thanks for your input, it’s what I was thinking, but the client secret is defined in a secret in the kubernetes cluster, and as far I could see it should imported as an env

envFromSecret: grafana-env-secret

am I right? did I miss something?

Thanks again.

jangaraj · April 18, 2024, 1:04pm

Ok, so what’s in that secret?

i5js · April 18, 2024, 2:11pm

Thanks again

apiVersion: v1
data:
  GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: <secret encoded in base64>
kind: Secret
metadata:
  creationTimestamp: null
  name: grafana-env-secret
  namespace: monitoring

I have logged to the pod and I can see the value:

KUBE_PROMETHEUS_STACK_PROMETHEUS_SERVICE_PORT_HTTP_WEB=9090
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=<secret uncoded>
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443

jangaraj · April 18, 2024, 2:46pm

Do you have special characters in client id/client secret?

i5js · April 18, 2024, 2:57pm

No, just capital and regular letters, I can send it by PM

jangaraj · April 18, 2024, 4:50pm

Is there any reverse proxy/waf in front of keycloak, which may block/encode/decode requests?

Try to exchange code for token manually (curl,…) - that’s just one request to token endpoint to verify that’s working fine outside grafana.

Why you just don’t switch to PKCE flow, so you don’t need to care about client credential at all. It just makes you a trouble.

i5js · April 18, 2024, 5:47pm

Yes, there a reverse proxy, nginx. How can I create the curl command? I’m sorry but I’m not very handy with that

Thanks

jangaraj · April 18, 2024, 5:58pm

Example:

curl -s -X POST \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode "client_id=${CLIENTID}" \
  --data-urlencode "client_secret=${CLIENTSECRET}" \
  --data-urlencode "grant_type=authorization_code" \
  --data-urlencode "code=${CODE}" \
  --data "redirect_uri=${REDIRECTURL}" \
  https://sso.home.domain.com/realms/home/protocol/openid-connect/token

This is where it is failing in your case - Grafana is not able to exchange code for a token, because Keycloak thinks that used client credentials are wrong. Search “Authorization Code Flow” for more details - how it works under the hood.

i5js · April 18, 2024, 6:17pm

What should I add here? ${CODE

jangaraj · April 18, 2024, 6:22pm

Code which is Grafana trying to exchange. Use browser console and watch requests/response. It will be there.

i5js · April 18, 2024, 6:40pm

I can’t find it, well, I don’t know how to find it.
It’s too much for me, and I have spent a lot of days on this error.

any doc to enable pkce auth in Grafana?

i5js · April 18, 2024, 6:49pm

I don’t know if it’s right, but I disabled Client authentication and it worked.

jangaraj · April 18, 2024, 6:50pm

use_pkce

i5js · April 18, 2024, 7:03pm

Added, but…

logger=context userId=0 orgId=0 uname= t=2024-04-18T19:01:00.810516491Z level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=10.244.0.0 time_ms=0 duration=592.658µs size=439 referer=https://monitoring.home.domain.com/grafana/login handler=/login/:name status_source=server
logger=authn.service t=2024-04-18T19:01:02.803233017Z level=error msg="Failed to authenticate request" client=auth.client.generic_oauth error="[auth.oauth.token.exchange] failed to exchange code to token: oauth2: \"unauthorized_client\" \"Invalid client or Invalid client credentials\""
logger=context userId=0 orgId=0 uname= t=2024-04-18T19:01:02.840936833Z level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=10.244.0.0 time_ms=101 duration=101.597105ms size=37 referer= handler=/login/:name status_source=server

jangaraj · April 18, 2024, 7:09pm

But that’s not just use_pkce: true - you need public client in the Keycloak, you need to remove client secret/GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET in Grafana, …

jangaraj · April 18, 2024, 7:30pm

You should mention what is “Client authentication”:

When client is public, then PKCE must be used.

i5js · April 18, 2024, 7:37pm

Ok, just one more thing

I was checking another service, and I’m using keycloak and oauth2-proxy to authenticate. It uses

  # Oauth2 client configuration. From Keycloak configuration
  clientID: "oauth2-proxy"
  clientSecret: "<code uncoded>"

And it works, perfectly, so perhaps it’s not the nginx-ingress the issue.

i5js · April 18, 2024, 7:43pm

It seems there is an opened issue:

Grafana - oauth keycloak environment variable issue · Issue #634 · grafana/helm-charts (github.com)