SSO - keycloak-grafana ( login.OAuthLogin(missing saved state )

Hi there,

as first one substitution, as only 2 links are available for new user:

LINK == https://xx.yy.z-zz.net/
DOMAIN == xx.yy.z-zz.net/

  • What Grafana version and what operating system are you using?

  • v7.5.8 (0184ed92b5) as deployments in k3s within otc cloud

  • What are you trying to achieve?
    use keycloak as SSO for grafana authentication

  • Can you copy/paste the configuration(s) that you are having problems with?
    grafana.ini: ( as configmap)

 grafana.ini: >
    [analytics]
    check_for_updates = true

    [auth]
    disable_login_form = false

    [auth.anonymous]
    enabled = true
    org_role = Viewer

    [auth.basic]
    enabled = false

    [dashboards]
    default_home_dashboard_path = /tmp/dashboards/rancher-default-home.json

    [grafana_net]
    url = grafanaDOTnet

    [log]
    mode = console file

    [paths]
    data = /var/lib/grafana/
    logs = /var/log/grafana
    plugins = /var/lib/grafana/plugins
    provisioning = /etc/grafana/provisioning

    [security]
    allow_embedding = true
    cookie_secure = true
    cookie_samesite = none
 
  
    [users]
    auto_assign_org_role = Viewer
    
    [server] 
    root_url = **LINK**
    domain =  **DOMAIN**


    [auth.generic_oauth]

    enabled = true 
    tls_skip_verify_insecure = false
    name = Keycloak 
    allow_sign_up = true 
    client_id = **DOMAIN**
    client_secret = xxxxxxxxxxxxxxxxxxxxxx 
    scopes =  roles
    auth_url =
    **LINK**/keycloak/realms/master/protocol/openid-connect/auth    

    token_url =
    **LINK**/keycloak/realms/master/protocol/openid-connect/token  

    api_url =
    **LINK**/keycloak/realms/master/protocol/openid-connect/userinfo 

    role_attribute_path = "contains(roles[*], 'admin') && 'Admin' || contains(roles[*], 'editor') && 'Editor' || 'Viewer'"

in keyclock:

clientID: DOMAIN
Client Protocol: openid-connect
accestype: confidential

standard flow: ON
directaccessgrantenabled: ON

root url: LINK
Valid Redirect URIs: LINK/login/generic_oauth
Base URL: /login/generic_oauth

roles: admin
mappers: i tried Client and also Realm roles

users: admin → role mappings: admin

so, i was able after editing grafana.ini to get button to keycloak, after click i am redirected to keycloak login page:

Request cookies:

|AUTH_SESSION_ID| a198f18f-baca-4f6a-89a8-6df71ff99f01
|AUTH_SESSION_ID_LEGACY| a198f18f-baca-4f6a-89a8-6df71ff99f01
|KC_RESTART| eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6IC

Response cookies:
KC_RESTART| eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6

, after type admin/password credential i stuck with:
Response cookies: → no values

also there are 2 items under headers names with errors:

error 302:
LINK/keycloak/realms/master/login-actions/authenticate?session_code=-6SnbUah3KOfW_3gmA_epkg_8VFjOxabkcNIAMD3a6I&execution=4d0592e9-cf80-45a3-99ec-369bfc3f2243&client_id=DOMAIN&tab_id=eI110lwrIeE

response headers:
LINK/login/generic_oauth?state=ToGwkt81MA5mAuiji8Yn0LZXeblG-a1WIfUh8cLuNOA=&session_state=a198f18f-baca-4f6a-89a8-6df71ff99f01&code=234f3b7c-51ba-4444-9a81-c138b7234203.a198f18f-baca-4f6a-89a8-6df71ff99f01.a1314508-1498-4209-a44d-3a26edde4f10

error 500:
LINK/login/generic_oauth?state=ToGwkt81MA5mAuiji8Yn0LZXeblG-a1WIfUh8cLuNOA=&session_state=a198f18f-baca-4f6a-89a8-6df71ff99f01&code=234f3b7c-51ba-4444-9a81-c138b7234203.a198f18f-baca-4f6a-89a8-6df71ff99f01.a1314508-1498-4209-a44d-3a26edde4f10

respons headers:
oauth_state=; Path=/; Max-Age=0; HttpOnly; Secure; SameSite=None

in tab cookies no value for resposne again…

Request cookies:
redirect_to %2F%3ForgId%3D1
oauth_state d937bc3f4644ad1fc487a235593f9356ebc0021459c4…

Response cookies:
oauth_state nothing

UI shows: login.OAuthLogin(NewTransportWithCode)

after refresh page

UI shows: login.OAuthLogin(missing saved state)

and in headers stays only error 500 wit no cookie again

Request cookies:
redirect_to %2F%3ForgId%3D1

Response cookies:
oauth_state nothing

i tried several changes within grafana.ini abut still nothing …

would you have any advice please?

Thanks a lot

one more thing:
when i put in address bar :
api_url = LINK:/keycloak/realms/master/protocol/openid-connect/userinfo
i got:
{"error":"invalid_request","error_description":"Token not provided"}

maybe update keycloak would help?
https://www.keycloak.org/2022/09/keycloak-1902-released

Make sure, you have correct cookie configuration for OIDC, e.g.:

GF_SECURITY_COOKIE_SECURE=true
GF_SECURITY_COOKIE_SAMESITE=none

Remember OIDC requires HTTPS.

BTW: I would use public client with PKCE (GF_AUTH_GENERIC_OAUTH_USE_PKCE=true), so you don’t need to manage a client secret.

Good afternoon,

well, i have mentioned settings in the grafana.ini:

[security]
    allow_embedding = true
    cookie_secure = true         <<<--------
    cookie_samesite = none   <<<--------

    [users]
    auto_assign_org_role = Viewer

    [server] 
    root_url = https://monitoring.campusedge.t-systems.net
    domain =  monitoring.campusedge.t-systems.net

    [auth.generic_oauth]
    use_pkce = true           <<<--------
    enabled = true 
    tls_skip_verify_insecure = false
    name = Keycloak 
    allow_sign_up = true 
    client_id = monitoring.xx.net 
    client_secret = xxxx
    scopes =  profile
    auth_url =    https://iam.xx.net/realms/master/protocol/openid-connect/auth    
    token_url =    https://iam.xx.net/realms/master/protocol/openid-connect/token  
    api_url =    https://iam.xx.net/realms/master/protocol/openid-connect/userinfo 
    role_attribute_path = "contains(roles[*], 'admin') && 'Admin' ||  contains(roles[*], 'editor') && 'Editor' || 'Viewer'"

but still nothing…

I’m using wildcard certificate for whole domain within ingress…

anyway, still same behavior…

im stuck :confused:

1 Like

Great. Do you have a test user? Could you provide har file of whole login procedure? Maybe max age =0 on the cookie is a problem - cookie will expire immediately. How ingress is configured?

good morning,

  • I’m using admin user.
  • sending har file ( complete procedure : load grafana index → login → keycloak login → add credential and login )

password: qwerty

  • ingress yaml file:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
  name: monitoring
  namespace: cattle-monitoring-system
spec:
  ingressClassName: nginx
  rules:
  - host: monitoring.DOMAIN.net
    http:
      paths:
      - backend:
          service:
            name: rancher-monitoring-grafana
            port:
              number: 80
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - monitoring.DOMAIN.net
    secretName: grafana-tls
status:
  loadBalancer:
    ingress:
    - ip: IP1
    - ip: IP2

-Maybe max age =0 on the cookie is a problem - cookie will expire immediately.

in “developer tools”-> console i can see;
Cookie “oauth_state” has been rejected because it is already expired.
settings of max age is a part of app or can I check somehow ?

  • two more things:
  1. when i put in the address bar userinfo link from the

Realm settings - > Endpoints → OpenID Endpoint Configuration
, i got:

{"error":"invalid_request","error_description":"Token not provided"}

  1. in logs:

k logs rancher-monitoring-grafana-6c8867db68-7tfxf -c grafana --follow

t=2022-10-20T11:54:16+0000 lvl=info msg=“HTTP Server Listen” logger=http.server address=[::]:3000 protocol=http subUrl= socket=

t=2022-10-20T11:55:00+0000 lvl=info msg=“Request Completed” logger=context userId=0 orgId=1 uname= method=GET path=/ status=302 remote_addr=10.0.1.129 time_ms=0 size=29 referer=“https://monitoring.DOMAIN.net/?orgId=1”

t=2022-10-20T11:55:08+0000 lvl=info msg=“Request Completed” logger=context userId=0 orgId=1 uname= method=GET path=/login/generic_oauth status=302 remote_addr=10.0.1.129 time_ms=0 size=358 referer=https://monitoring.DOMAIN.net/login

t=2022-10-20T11:55:17+0000 lvl=info msg=“state check” logger=oauth queryState=879fc67ac97297f51893ba5c6f65d78f2460d6f2b083518e925a10867cc785cb cookieState=879fc67ac97297f51893ba5c6f65d78f2460d6f2b083518e925a10867cc785cb
t=2022-10-20T11:55:17+0000 lvl=eror msg=login.OAuthLogin(NewTransportWithCode) logger=context userId=0 orgId=1 uname= error=“Post "https://iam.DOMAIN.net/realms/master/protocol/openid-connect/token\”: x509: certificate signed by unknown authority"

t=2022-10-20T11:55:17+0000 lvl=eror msg=“Request Completed” logger=context userId=0 orgId=1 uname= method=GET path=/login/generic_oauth status=500 remote_addr=10.0.1.129 time_ms=92 size=1744 referer=

im using as i said, wildcard certificate for diff app in environment (grafana,rancher…) …

Thanks a lot.

Use:

[auth.generic_oauth]
    tls_skip_verify_insecure = true

to “fix” x509: certificate signed by unknown authority error. (Solution is to add used CA cert, so pod can verify tls).

For expired cookie replicate the same setup locally (without any proxy) and check if cookie there is also not valid. If not, then blame your infrastructure in front of Grafana for cookie manipulation.

Good morning,

so ,

i disabled tls check:

    tls_skip_verify_insecure = true

it does not work… in logs x509 error disappeared but got info :

Error getting email address" logger=oauth.generic_oauth url=https://iam.DOMAIN.net/realms/master/protocol/openid-connect/userinfo/emails error="{\"error\":\"RESTEASY003210: Could not find resource for full path: https://iam.DOMAIN.net/realms/master/protocol/openid-connect/userinfo/emails\"}"

as i already faced, but i search and find:
so i add email address for user in keycloak and I’m able to login inside. O_o

When i enable tls check , I’m getting x509 error again.

so , i was looking and find :

but no idea how to add my cert for the “backend” , don’t ask for solution, just point somewhere …

Thanks

Are you sure that you used

api_url =    https://iam.xx.net/realms/master/protocol/openid-connect/userinfo

Do you have recent Grafana version?

I guess you are using Rancher, so Rancher doc may give you hint how to use custom CA certificate, e.g. About Custom CA Root Certificates | Rancher Manager

version is: Grafana v7.5.8 (0184ed92b5
maybe also update keycloak is worth for it

yes, info from OpenID Endpoint Configuration:

https://iam.xx.net/realms/master/.well-known/openid-configuration

"userinfo_endpoint":"https://iam.xx.net/realms/master/protocol/openid-connect/userinfo"

yep, using rancher , but we have moved from titanium cloud to harvester

thanks a lot

@jangaraj

so, i tried import tls-additional-ca into rancher but no affect.

i found on your advice ( here :

Build own crt file with all required CA certs and mount it to the /etc/ssl/certs/ca-certificates.crt (file location is valid for docker images based on the Debian) in the Grafana container.

now, when i enable tls check [ tls_skip_verify_insecure = false ] in grafana.ini , i do not get x509 error and login works.

logs:

t=2022-10-25T09:59:07+0000 lvl=info msg="state check" logger=oauth queryState=9229d03422f27de2917f1641c5d8004648bf3832a3051ec18756e5796cdd09a8 cookieState=9229d03422f27de2917f1641c5d8004648bf3832a3051ec18756e5796cdd09a8
t=2022-10-25T09:59:07+0000 lvl=info msg="Successful Login" logger=http.server User=admin@admin.com
t=2022-10-25T09:59:07+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=1 uname= method=GET path=/login/generic_oauth status=302 remote_addr=10.0.1.129 time_ms=81 size=32 referer=  

but,
I logged into pod grafana and check file /etc/ssl/certs/ca-certificates.crt if it contains my certificate, but I could not find it…is there something what is missing to me ?

Thank You