Connection to Keycloak broken, possibly due to a Grafana upgrade

tomz · November 9, 2022, 1:32pm

Due to an unplanned update of my Grafana Docker image I ended up with a new version of Grafana which seems to have somehow broken my Oauth connection to Keycloak.
I accidentally have been using grafana/grafana as image, so I don’t know what my previous version was, but I am now on v2.9.3

Steps to reproduce:

Select "Login with Keycloak on Grafana login page
Authenticate in Keycloak
Redirect leads to error message below

Below is the error I get in the grafana logs:

logger=oauth.generic_oauth t=2022-11-09T07:53:48.825034947Z level=error msg="Error getting email address" url=https://iam.myDomain.com/realms/myDomain/protocol/openid-connect/userinfo/emails error="{\"error\":\"RESTEASY003210: Could not find resource for full path: https://iam.myDomain.com/realms/myDomain/protocol/openid-connect/userinfo/emails\"}"

All users have got proper email addresses in Keycloak.
Were there any recent changes in the OAuth section of Grafana which may have led to a change in behaviour?

I realise that the Grafana update may be a false flag, and whatever the issue is, was simply exposed by restarting the server, but right now it’s the only change to Grafana and Keycloak I can think of.

Some more detail in case it’s relevant:
docker-compose.yml section:

grafana:
    image: grafana/grafana
    container_name: grafana
    ports:
      - 3000:3000
    restart: unless-stopped
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: myPassword

      GF_SERVER_DOMAIN: "grafana.myDomain.com"
      GF_SERVER_ROOT_URL: "https://grafana.myDomain.com"
      GF_AUTH_GENERIC_OAUTH_ENABLED: "true"
      GF_AUTH_GENERIC_OAUTH_NAME: "SingleSignOn"
      GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP: "true"
      GF_AUTH_GENERIC_OAUTH_CLIENT_ID: "Grafana"
      GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: "mySecret"
      GF_AUTH_GENERIC_OAUTH_SCOPES: profile
      GF_AUTH_GENERIC_OAUTH_AUTH_URL: "https://iam.myDomain.com/realms/myDomain/protocol/openid-connect/auth"
      GF_AUTH_GENERIC_OAUTH_TOKEN_URL: "https://iam.myDomain.com/realms/myDomain/protocol/openid-connect/token"
      GF_AUTH_GENERIC_OAUTH_API_URL: "https://iam.myDomain.com/realms/myDomain/protocol/openid-connect/userinfo"
      GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH: "contains(roles[*], 'admin') && 'Admin' || contains(roles[*], 'editor') && 'Editor' || 'Viewer'"

Keycloak version is Version 19.0.1, I tried v20.0.1 as well without success.

Thanks
Thomas

jangaraj · November 9, 2022, 3:46pm

BTW: You will have unplanned upgrades again with this definition. Use version tag explicitly (e.g. image: grafana/grafana:9.2.3).

See how email is detected:

Make sure, that Grafana will find email in step 1-3, because your IDP apparently doesn’t support step 4.

tomz · November 9, 2022, 4:15pm

Oh I know, have been bitten by this with other containers. Somehow this one survived unnoticed
Thanks for the link, I’ll work through it and will report back. The puzzling piece is that my setup has been working flawless for a few months and only broke after this update. Well, we’ll see…

tomz · November 10, 2022, 1:37pm

I made a little bit of progress, if I understand things right, the link you pointed me to basically says the email address needs to be provided in the OAuth token.
So I went ahead and deciphered what Keycloak is returing via calling it directly from the cli:

Followed by sticking the token into this service JSON Web Tokens - jwt.io
At first there was indeed no email, nowhere. I therefore updated the client mapper in Keycloak, adding the email address.
Following that I can see that the email is now part of the payload:

Is this where it should be?
In any case, the error message didn’t change. Grafana still can’t find the email.

Edit:

Check for the presence of an e-mail address using the JMESPath specified via the email_attribute_path configuration option

Right now the JMESPAth in my settings looks like this:

GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH: “contains(roles[], ‘admin’) && ‘Admin’ || contains(roles[], ‘editor’) && ‘Editor’ || ‘Viewer’”

The statement above sounds like I need to somehow add email_attribute_path to this string so the email address is extracted?

Edit2:
Read up some more, and now believe that what I have to do is to add this

GF_AUTH_GENERIC_OAUTH_EMAIL_ATTRIBUTE_NAME: “email”

to my docker-compose.yml to match the new “email” field in the screenshot higher up.
Unfortunately no difference.
There’s also GF_AUTH_GENERIC_OAUTH_EMAIL_ATTRIBUTE_PATH which may play a role here, but I am not sure if I need it and what it should look like.

jangaraj · November 10, 2022, 2:36pm

Increase Grafana log level and you will see everything in the logs (and you won’t need insecure direct access grant to “see” a token).

tomz · November 10, 2022, 3:09pm

Switched to debug level, and here’s something:

No id_token found" token="unsupported value type"

Not sure if this means that a value named “type” is wrong, or if one of the many “value-types” is off. If it’s the later I guess I need to somehow find out which value-type is causing the issue. But definitely a start.

grafana  | logger=oauth.generic_oauth t=2022-11-10T14:52:15.789642588Z level=debug msg="Getting user info"
grafana  | logger=oauth.generic_oauth t=2022-11-10T14:52:15.789652226Z level=debug msg="Extracting user info from OAuth token"
grafana  | logger=oauth.generic_oauth t=2022-11-10T14:52:15.789659908Z level=debug msg="No id_token found" token="unsupported value type"
grafana  | logger=oauth.generic_oauth t=2022-11-10T14:52:15.789670245Z level=debug msg="Getting user info from API"
grafana  | logger=oauth.generic_oauth t=2022-11-10T14:52:15.793005501Z level=debug msg="Error getting user info from API" url=https://iam.mydomain.com/realms/myRealm/protocol/openid-connect/userinfo error=…

I searched through the token clear text and decoded, but there is no “type” anywhere. Only “typ”: “Bearer” in the decoded payload would come closest. A typo? Most likely not…

My knowledge in this area is pretty shallow so I am trying to make sense of this all.
The beginning of the payload is:

grafana | logger=oauth t=2022-11-10T14:52:15.7895974Z level=debug msg=“OAuthLogin: got token” token="&{AccessToken:eyJhb…

So I got a token of type AccessToken, and at the end of the payload I can find this

…e7433cf8d6 token_type:Bearer]}"

But Grafana seems to expect an id_token. Is Keycloak sending the wrong token type?

msg=“No id_token found” token=“unsupported value type”

jangaraj · November 10, 2022, 3:21pm

I bet you don’t have id token, because you didn’t use openid scope, so you have pure OAuth (and not OpenID Connect). Configure GF_AUTH_GENERIC_OAUTH_SCOPES properly.

tomz · November 10, 2022, 5:41pm

I am afraid, I don’ know what a proper configuration of the scopes would look like.

Before things stopped working I had just “profile” as scope in Grafana.

GF_AUTH_GENERIC_OAUTH_SCOPES: “profile”

I meanwhile changed the email scope in Keycloak from optional to default, which makes the email address show up in the token payload.
I tried adding “email” explicitly to the scope, with and without “profile”, but no success.

GF_AUTH_GENERIC_OAUTH_SCOPES: “profile, email”

Other than that, the client is set up as OpenID connect (#1)

jangaraj · November 10, 2022, 7:13pm

openid scope must be requested:

GF_AUTH_GENERIC_OAUTH_SCOPES=openid profile email

tomz · November 10, 2022, 7:19pm

Bingo! Worked right away.
Thank you very much for helping me along. I definitely learned something new along the way.

jangaraj · November 10, 2022, 7:40pm

I would recommend to use also PKCE flow (public client on the Keycloak side):

GF_AUTH_GENERIC_OAUTH_USE_PKCE=true

So you don’t need to manage a client secret - (GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET) on the Grafana side - that will be a standard security in 2022.

Also change your password pls - you exposed your Keycloak URL, login + also your password partially.

tomz · November 10, 2022, 8:10pm

Oh dear, the downside of screenshots. All good advice! Thanks again

guogangfan · August 8, 2023, 3:49am

@jangaraj I have same issue, while openid scope already include. Can you help ?

Keycloak version: 22.0.1
Grafana Version: 10.0.3 (installed by kube-prometheus-stack helm chart)

my configs:

grafana:
  persistence:
    enabled: true
    storageClassName: "rook-ceph-block"
  grafana.ini:
    server:
      domain: grafana.xxx.com
      root_url: https://grafana.xxx.com
    log:
      filters: "oauth.generic_oauth:debug"
    auth.generic_oauth:
      enabled: true
      name: Keycloak
      client_id: grafana
      client_secret: gLfARsNXIIxb0eK5Xj9nnb8G6UKIITt1
      scopes: ["openid", "profile", "email", "offline_access", "roles"]
      role_attribute_path: "contains(roles[*], 'grafana_admin') && 'Admin' || contains(roles[*], 'grafana_editor') && 'Editor' || 'Viewer'"
      email_attribute_path: "email"
      login_attribute_path: "username"
      name_attribute_path: "full_name"
      auth_url: "https://keycloak.xxx.com/realms/itps/protocol/openid-connect/auth"
      token_url: "http://keycloak.security.svc.cluster.local/realms/itps/protocol/openid-connect/token"
      # Note: api_url is not required if the id_token contains all the necessary user information and can add latency to the login process. 
      # It is useful as a fallback or if the user has more than 150 group memberships
      api_url: "http://keycloak.security.svc.cluster.local/realms/itps/protocol/openid-connect/userinfo"
      allow_sign_up: true
    panels:
      disable_sanitize_html: true

logs:

logger=context userId=0 orgId=0 uname= t=2023-08-08T03:45:17.44805103Z level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=172.16.89.0 time_ms=1 duration=1.071894ms size=267 referer=https://grafana.bgzchina.com/login handler=/login/:name
logger=oauth t=2023-08-08T03:45:27.417034569Z level=info msg="state check" queryState=922a0fe801fb1ed0fb8937d9cf3f9a0c86c87cc8ed137cf0f1809b443675a499 cookieState=922a0fe801fb1ed0fb8937d9cf3f9a0c86c87cc8ed137cf0f1809b443675a499
logger=oauth.generic_oauth t=2023-08-08T03:45:27.428978661Z level=debug msg="Getting user info"
logger=oauth.generic_oauth t=2023-08-08T03:45:27.429050436Z level=debug msg="Extracting user info from OAuth token"
logger=oauth.generic_oauth t=2023-08-08T03:45:27.429060039Z level=debug msg="No id_token found" token="unsupported value type"
logger=oauth.generic_oauth t=2023-08-08T03:45:27.429081616Z level=debug msg="Getting user info from API"
logger=oauth.generic_oauth t=2023-08-08T03:45:27.431844892Z level=debug msg="Error getting user info from API" url=http://keycloak.security.svc.cluster.local/realms/itps/protocol/openid-connect/userinfo error=
logger=oauth.generic_oauth t=2023-08-08T03:45:27.432880616Z level=error msg="Error getting email address" url=http://keycloak.security.svc.cluster.local/realms/itps/protocol/openid-connect/userinfo/emails error="{\"error\":\"RESTEASY003210: Could not find resource for full path: http://keycloak.security.svc.cluster.local/realms/itps/protocol/openid-connect/userinfo/emails\"}"
logger=context userId=0 orgId=0 uname= t=2023-08-08T03:45:27.432956841Z level=error msg="login.OAuthLogin(get info from generic_oauth)" error="Error getting email address: {\"error\":\"RESTEASY003210: Could not find resource for full path: http://keycloak.security.svc.cluster.local/realms/itps/protocol/openid-connect/userinfo/emails\"}"