Connection to Keycloak broken, possibly due to a Grafana upgrade

Due to an unplanned update of my Grafana Docker image I ended up with a new version of Grafana which seems to have somehow broken my Oauth connection to Keycloak.
I accidentally have been using grafana/grafana as image, so I don’t know what my previous version was, but I am now on v2.9.3

Steps to reproduce:

  1. Select "Login with Keycloak on Grafana login page
  2. Authenticate in Keycloak
  3. Redirect leads to error message below

image

Below is the error I get in the grafana logs:

logger=oauth.generic_oauth t=2022-11-09T07:53:48.825034947Z level=error msg="Error getting email address" url=https://iam.myDomain.com/realms/myDomain/protocol/openid-connect/userinfo/emails error="{\"error\":\"RESTEASY003210: Could not find resource for full path: https://iam.myDomain.com/realms/myDomain/protocol/openid-connect/userinfo/emails\"}"

All users have got proper email addresses in Keycloak.
Were there any recent changes in the OAuth section of Grafana which may have led to a change in behaviour?

I realise that the Grafana update may be a false flag, and whatever the issue is, was simply exposed by restarting the server, but right now it’s the only change to Grafana and Keycloak I can think of.

Some more detail in case it’s relevant:
docker-compose.yml section:

grafana:
    image: grafana/grafana
    container_name: grafana
    ports:
      - 3000:3000
    restart: unless-stopped
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: myPassword

      GF_SERVER_DOMAIN: "grafana.myDomain.com"
      GF_SERVER_ROOT_URL: "https://grafana.myDomain.com"
      GF_AUTH_GENERIC_OAUTH_ENABLED: "true"
      GF_AUTH_GENERIC_OAUTH_NAME: "SingleSignOn"
      GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP: "true"
      GF_AUTH_GENERIC_OAUTH_CLIENT_ID: "Grafana"
      GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: "mySecret"
      GF_AUTH_GENERIC_OAUTH_SCOPES: profile
      GF_AUTH_GENERIC_OAUTH_AUTH_URL: "https://iam.myDomain.com/realms/myDomain/protocol/openid-connect/auth"
      GF_AUTH_GENERIC_OAUTH_TOKEN_URL: "https://iam.myDomain.com/realms/myDomain/protocol/openid-connect/token"
      GF_AUTH_GENERIC_OAUTH_API_URL: "https://iam.myDomain.com/realms/myDomain/protocol/openid-connect/userinfo"
      GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH: "contains(roles[*], 'admin') && 'Admin' || contains(roles[*], 'editor') && 'Editor' || 'Viewer'"

Keycloak version is Version 19.0.1, I tried v20.0.1 as well without success.

Thanks
Thomas

BTW: You will have unplanned upgrades again with this definition. Use version tag explicitly (e.g. image: grafana/grafana:9.2.3).

See how email is detected:

Make sure, that Grafana will find email in step 1-3, because your IDP apparently doesn’t support step 4.

Oh I know, have been bitten by this with other containers. Somehow this one survived unnoticed :slight_smile:
Thanks for the link, I’ll work through it and will report back. The puzzling piece is that my setup has been working flawless for a few months and only broke after this update. Well, we’ll see…

I made a little bit of progress, if I understand things right, the link you pointed me to basically says the email address needs to be provided in the OAuth token.
So I went ahead and deciphered what Keycloak is returing via calling it directly from the cli:

Followed by sticking the token into this service JSON Web Tokens - jwt.io
At first there was indeed no email, nowhere. I therefore updated the client mapper in Keycloak, adding the email address.
Following that I can see that the email is now part of the payload:

Is this where it should be?
In any case, the error message didn’t change. Grafana still can’t find the email.

Edit:

Check for the presence of an e-mail address using the JMESPath specified via the email_attribute_path configuration option

Right now the JMESPAth in my settings looks like this:

GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH: “contains(roles[], ‘admin’) && ‘Admin’ || contains(roles[], ‘editor’) && ‘Editor’ || ‘Viewer’”

The statement above sounds like I need to somehow add email_attribute_path to this string so the email address is extracted?

Edit2:
Read up some more, and now believe that what I have to do is to add this

GF_AUTH_GENERIC_OAUTH_EMAIL_ATTRIBUTE_NAME: “email”

to my docker-compose.yml to match the new “email” field in the screenshot higher up.
Unfortunately no difference.
There’s also GF_AUTH_GENERIC_OAUTH_EMAIL_ATTRIBUTE_PATH which may play a role here, but I am not sure if I need it and what it should look like.

Increase Grafana log level and you will see everything in the logs (and you won’t need insecure direct access grant to “see” a token).

1 Like

Switched to debug level, and here’s something:

No id_token found" token="unsupported value type"

Not sure if this means that a value named “type” is wrong, or if one of the many “value-types” is off. If it’s the later I guess I need to somehow find out which value-type is causing the issue. But definitely a start.

grafana  | logger=oauth.generic_oauth t=2022-11-10T14:52:15.789642588Z level=debug msg="Getting user info"
grafana  | logger=oauth.generic_oauth t=2022-11-10T14:52:15.789652226Z level=debug msg="Extracting user info from OAuth token"
grafana  | logger=oauth.generic_oauth t=2022-11-10T14:52:15.789659908Z level=debug msg="No id_token found" token="unsupported value type"
grafana  | logger=oauth.generic_oauth t=2022-11-10T14:52:15.789670245Z level=debug msg="Getting user info from API"
grafana  | logger=oauth.generic_oauth t=2022-11-10T14:52:15.793005501Z level=debug msg="Error getting user info from API" url=https://iam.mydomain.com/realms/myRealm/protocol/openid-connect/userinfo error=…

I searched through the token clear text and decoded, but there is no “type” anywhere. Only “typ”: “Bearer” in the decoded payload would come closest. A typo? Most likely not…

My knowledge in this area is pretty shallow so I am trying to make sense of this all.
The beginning of the payload is:

grafana | logger=oauth t=2022-11-10T14:52:15.7895974Z level=debug msg=“OAuthLogin: got token” token="&{AccessToken:eyJhb…

So I got a token of type AccessToken, and at the end of the payload I can find this

…e7433cf8d6 token_type:Bearer]}"

But Grafana seems to expect an id_token. Is Keycloak sending the wrong token type?

msg=“No id_token found” token=“unsupported value type”

I bet you don’t have id token, because you didn’t use openid scope, so you have pure OAuth (and not OpenID Connect). Configure GF_AUTH_GENERIC_OAUTH_SCOPES properly.

I am afraid, I don’ know what a proper configuration of the scopes would look like.

Before things stopped working I had just “profile” as scope in Grafana.

GF_AUTH_GENERIC_OAUTH_SCOPES: “profile”

I meanwhile changed the email scope in Keycloak from optional to default, which makes the email address show up in the token payload.
I tried adding “email” explicitly to the scope, with and without “profile”, but no success.

GF_AUTH_GENERIC_OAUTH_SCOPES: “profile, email”

Other than that, the client is set up as OpenID connect (#1)

openid scope must be requested:

GF_AUTH_GENERIC_OAUTH_SCOPES=openid profile email

Bingo! Worked right away.
Thank you very much for helping me along. I definitely learned something new along the way.

I would recommend to use also PKCE flow (public client on the Keycloak side):

GF_AUTH_GENERIC_OAUTH_USE_PKCE=true

So you don’t need to manage a client secret - (GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET) on the Grafana side - that will be a standard security in 2022.

Also change your password pls - you exposed your Keycloak URL, login + also your password partially.

Oh dear, the downside of screenshots. All good advice! Thanks again