Missing state cookie with OAuth2: but the oauth_state cookie is sent

Grafana Authentication
1

Hi,

I just spend a few hours trying to find out why Grafana can’t accept login from Google OAuth2 authentication.

Grafana complains about not finding the oauth_state cookie at the end of the oAuth tunnel (/login/google?state=(...). But the cookie is sent !

Usually I know how to investigate this kind of problem… but not today because everything looks clean and the cookie is sent by the browser. I’m looking for new ideas !

Step by step description:

  • Grafana send the user to https://accounts.google.com/signin/oauth/consent?(...), great.

  • The authorisation tunnel is doing it’s job

  • google redirect to https://mydomain.com/login/google?state=(...). The browser provides two cookies oauth_state and oauth_code_verifier affected to the right domain (no problem with samesite policy, it would be too easy !).

  • the endpoint /login/google logs:


[auth.oauth.state.missing] missing state cookie

and send the user to the /login page with a 302 :frowning:

Context

  • Grafana dockerized grafana/grafana:11.2.2, no plugin, minimal config.

  • Deployed on cloudrun, can be accessed by mydomain.com

  • No error, nothing strange before the endpoint /login/google

Everything work in localhost:xxxx

Note that when I run the same docker image locally, everything is fine, grafana find the cookie.

(localhost is added to Authorized JavaScript origins and redirect uri along side the real domain, GF_SERVER_ROOT_URL is ajusted)

Grafana config


[security]

disable_initial_admin_creation = true

; admin_user = admin

cookie_samesite = lax

[auth.basic]

disable_login_form = true

enabled = false

[auth.google]

enabled = true

allow_sign_up = true

auto_login = false

client_id = CLIENT_ID

client_secret = CLIENT_SECRET

scopes = openid email profile

auth_url = https://accounts.google.com/o/oauth2/v2/auth

token_url = https://oauth2.googleapis.com/token

api_url = https://openidconnect.googleapis.com/v1/userinfo

allowed_domains = alfred-eyes.ai

hosted_domain = alfred-eyes.ai

use_pkce = true

use_refresh_token = true

skip_org_role_sync = false

role_attribute_path = "'Editor'"

also theses env var are set

  • GF_AUTH_GOOGLE_CLIENT_ID

  • GF_AUTH_GOOGLE_CLIENT_SECRET

  • GF_SERVER_ROOT_URL

I don’t know where to look at from here.

The cookie tab for /login/google?state=(…)

image
image1694×460 65.3 KB

image
image2362×1002 311 KB

2

Increase log level and check debug server logs.

3

thanks for the suggestion ! Log level increased to debug

unfortunately, same information [auth.oauth.state.missing] missing state cookie

logger=authn.service traceID=1a803a72c23a1de821ca7024d4a5d60d t=2024-10-03T14:18:30.510372798Z level=info msg="Failed to authenticate request" client=auth.client.google error="[auth.oauth.state.missing] missing state cookie"

image
image2464×1078 304 KB

… any other idea ?

4

Those aren’t debug logs. You just showed a two log lines of info level.

5

Log level is set to debug’ in grafana.ini. is there a way to to configure it differently ?

Or is it about quantity? I will provide you more log soon.

I’m checking out the code of grafana… There is just one way to get the missing state cookie error. The
cookie is just not there.

  • chrome console don’t lie: the cookie is sent
  • cloud run doesn’t touch cookie

could a go middleware eat cookies ? I’m out of ideas

6

Did you restart Grafana to apply config change? Did you set log level in the right config section?

7

Thank you for your attention,

grafana instance is started each time I deploy.

[log]
level = debug

Here an extract of the log of Grafana. You can search for the word missing, and yes there is a lot of level=debug

https://file.io/D6axHCnyUcNE

It’s sad that the bug could not be reproduced on my machine, I could have run it with a debugger, just to confirm that the cookie is not there, despite what chrome pretend to send.

8

Dear jjangaraj,
I just found the cookie eater: it’s this feature of firebase hosting that redirect traffic to cloud run.

I don’t know exactly how yet but when I log using an instance of grafana that do not use firebase hosting, it works !

So the problem has nothing to do with Grafana, but I will update you just in case it could help. other people.

Thanks again for your attention.

1 Like
9

Final story, TLDR: Do not use firebase hosting with Grafana.

firebase hosting is great, but only for single page app that rely on http other headers (Authorisation…).
Grafana is not a single page app, it rely on cookies and sessions.

When using Firebase Hosting together with Cloud Functions or Cloud Run, cookies are generally stripped from incoming requests
(source)

Voila !