-
What Grafana version and what operating system are you using?
Grafana 11.0.0 using whatever OS comes with the grafana-enterprise:11.0.0 image.
-
What are you trying to achieve?
I’m upgrading from Grafana 9.3.2 to 11.0.0. We use Okta for oauth and alongside this upgrade I’d like to move from generic_oauth to okta auth in our config settings. Despite being the same actual oauth provider, to Grafana it appears to be a different one since the config will be a bit different.
-
How are you trying to achieve it?
I deployed a new grafana 11 image to our k8s test setup and it automatically migrated the db it uses.
-
What happened?
Oauth is broken unless I turn on oauth_allow_insecure_email_lookup which I don’t really want to do. On the login page I see “Login failed User sync failed” and in the logs I see
grafana logger=user.sync t=2024-06-14T15:38:22.564102069Z level=error msg="Failed to create user" error="user already exists" auth_module=oauth_okta auth_id=...
grafana logger=authn.service t=2024-06-14T15:38:22.564145609Z level=error msg="Failed to run post auth hook" client=auth.client.okta id= error="[user.sync.internal] unable to create user: user already exists"
When I do enable oauth_allow_insecure_email_lookup I am able to log in but user permissions get mangled. All admin users get set back to viewer role when they log in.
I would like to do a one-off migration from the old oauth provider to the new without allowing insecure email lookup and while also preserving user permissions.
GF_AUTH_OKTA_ENABLED: "true"
GF_AUTH_OKTA_SCOPES: "openid profile email"
GF_AUTH_OKTA_AUTH_URL: "https://redacted.okta.com/oauth2/v1/authorize"
GF_AUTH_OKTA_TOKEN_URL: "https://redacted.okta.com/oauth2/v1/token"
GF_AUTH_OKTA_API_URL: "https://redacted.okta.com/oauth2/v1/userinfo"
GF_AUTH_OKTA_CLIENT_ID: redacted
GF_AUTH_OKTA_CLIENT_SECRET: redacted
Here are the old settings:
GF_AUTH_GENERIC_OAUTH_NAME: okta
GF_AUTH_GENERIC_OAUTH_ENABLED: "true"
GF_AUTH_GENERIC_OAUTH_SCOPES: "openid profile email"
GF_AUTH_GENERIC_OAUTH_AUTH_URL: "https://redacted.okta.com/oauth2/v1/authorize"
GF_AUTH_GENERIC_OAUTH_TOKEN_URL: "https://redacted.okta.com/oauth2/v1/token"
GF_AUTH_GENERIC_OAUTH_API_URL: "https://redacted.okta.com/oauth2/v1/userinfo"
GF_AUTH_GENERIC_OAUTH_CLIENT_ID: redacted
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: redacted
- Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
On the login page I see “Login failed User sync failed” and in the logs I see
grafana logger=user.sync t=2024-06-14T15:38:22.564102069Z level=error msg="Failed to create user" error="user already exists" auth_module=oauth_okta auth_id=...
grafana logger=authn.service t=2024-06-14T15:38:22.564145609Z level=error msg="Failed to run post auth hook" client=auth.client.okta id= error="[user.sync.internal] unable to create user: user already exists"
I would stop Grafana and remove OAuth users from users table manually. Of course: backup, backup, backup and then test.
So new auth method will create users from the scratch. Probably you will loose user ids, so dashboard histories may be weird for a moment.
So are you saying to truncate the user table in the DB?
No, delete only oauth users. There can be also non oauth users, so truncate won’t be very smart.
So then are you saying to remove all entries in the user table which have an entry in the user_auth table with auth_module set to oauth_generic_oauth?
Either way, won’t removing users also remove their permissions? My hope is to be able to retain user permissions.
update user_auth table by renaming all users withusername_old
something like this?
UPDATE public.user SET name = concat(name, " OLD");
For reference, the fields look something like this
id login email name
2 caleb.woofenden@org.com caleb.woofenden@org.com Caleb Woofenden
not user table, user_auth right? or wait…
you need the user because you need their perms and all…
The user_auth table does not have a field for username
but user auth has fk relationship to user.id
in fact tons of tables have fk relationship to user.id. be careful
So are you saying I should update the name field in user? What does this accomplish also - does this just mean new users will be created for each person who logs in after I do this?
that is what you need to avoid, otherwise if new users are created then they all lose permissions because they are given nee user.id unless you do post cleanup which is messy
so when you implement this new auth what changes in user table.
to find out spin up a new test grafana, restore the grafana backend and do the switch and see what happens
OK, then you have to spend a time to understand internal DB model on your own. Is it really worth it?
When I set GF_AUTH_OAUTH_ALLOW_INSECURE_EMAIL_LOOKUP: "true" I am able to log in with the new Okta config. A new entry is added in the user_auth table for my user ID with auth_module field set to oauth_okta. The old entry for my user with oauth_generic_oauth is still present. In the org_user table, the entry for my user ID has its role changed from Admin to Viewer.
To reiterate, my hopes here are:
- To not have to set
GF_AUTH_OAUTH_ALLOW_INSECURE_EMAIL_LOOKUP: "true" to be able to log in. I’d rather our grafana not use something insecure 
- To not have my role set to
Viewer when I log in for the first time with the new oauth config. I’d like it to stay what it was: Admin.
I found a process that “works” but is really not ideal:
- Take a snapshot of the
org_user table to have a record of user roles
- Update the Oauth settings and set
GF_AUTH_OAUTH_ALLOW_INSECURE_EMAIL_LOOKUP: "true"
- Tell every user to log in (This is where the problem is. I can’t be blocked waiting for every user to log in.)
- They will then get a new
user_auth row with the new oauth config
- Their
role in org_user will be set to Viewer
- Their user ID will be unchanged (This is great)
- Run a query that updates all
org_user rows to have the same role as before
- Set
GF_AUTH_OAUTH_ALLOW_INSECURE_EMAIL_LOOKUP: "false"
I’ve tested this process with my own user and I am still able to log in after setting GF_AUTH_OAUTH_ALLOW_INSECURE_EMAIL_LOOKUP: "false" which is the goal. I just don’t think I can actually do this process unless I can initiate login attempts from every user myself.
what setting was it with previous auth? and how did it allow login
is there no documentation on migrating to a new auth?
what is your new user data look like when you login first time, is the row identical other than user.id?
Correct me if I’m wrong, but I think GF_AUTH_OAUTH_ALLOW_INSECURE_EMAIL_LOOKUP was not an option on 9.3.2 which is what we’re coming from. Either that or it defaults to true in 9.3.2.
I haven’t been able to find any documentation on migrating to a different oauth provider (even though ours is the same; this is just a migration from using generic_oauth to okta in the grafana settings).
user data is not changed by this at all. Only user_auth and org_user is:
A new entry is added in the user_auth table for my user ID with auth_module field set to oauth_okta . The old entry for my user with oauth_generic_oauth is still present. In the org_user table, the entry for my user ID has its role changed from Admin to Viewer .
I would file that as a bug unless it is a feature 
to prevent the security issues linked above by @jangaraj
user data - intact
user_auth data -
old entry intact with auth_module = oauth_generic_oauth
new entry created with auth_module = oauth_okta
org_user - updated from Admin to Viewer.
Ah thanks, I did read the first one earlier this week but in both cases the only recommended action is to use the insecure emails setting. I don’t see any documentation on migrating users to a different oauth provider while disabling that insecure setting.
