Issues with Grafana after Migration to PostgreSQL and Upgrade

Grafana Dashboards
, , ,
1
  • What Grafana version and what operating system are you using?

Grafana 10.xx
but downgraded to 9.4

  • What are you trying to achieve?
    Recently, we made significant changes to our Grafana configuration. We upgraded from version 9.x.x to 10.x.x and migrated our background database from SQLlite to PostgreSQL to support a high-availability (HA) setup for our production environment.

Initially, everything seemed fine, but we started facing problems with user logins. New users are unable to log in, while existing users in the users table can still access their accounts. Our authentication process relies on Azure AD without multitenant architecture.

The specific errors reported by users include “user_pkey1 violation” and “user already exists.” We attempted to address the issue by setting “oauth_allow_insecure_email_lookup” to true, but unfortunately, this did not resolve the problem.

image
image719Ă—316 18.7 KB

image
image720Ă—378 19.8 KB

image

Aside from the login issues, we are experiencing difficulties with modifying existing dashboards and creating new ones. All attempts to save modified dashboards result in “failed to save dashboard” errors.

In an attempt to rectify the situation, we referred to bug reports on GitHub and decided to roll back Grafana to version 9.4.x, which partially fixed the dashboard-saving issue for some users. However, some users are still facing the same problem, and no new dashboards can be created. Additionally, new users remain unable to log in.

These issues are causing significant disruptions to our team, and we would greatly appreciate your valuable insights and suggestions on possible fixes to resolve these problems.

  • How are you trying to achieve it?

  • What happened?

  • What did you expect to happen?

  • Can you copy/paste the configuration(s) that you are having problems with?

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

  • Did you follow any online instructions? If so, what is the URL?

image

2

I would delete that user(s) with “user already exists.” error, so they will be recreated from the scratch.

3

We tried the same …But 1st time they are getting user_pkey1 error then for the 2nd time they get user already exists error…(No login is happening for these new users )Tried couple of times but no luck .

4

Was this migration fully tested in a staging or qa environment or was the upgrade done on a live production system?

5

Yes .Initially we tested it in test env and did basic functionality testing which seems working .Then we went and did this migration on production .Even this migration was fine until people started complaining about these one after another .

1 Like
6

So did you have uat tests, as in actual users loggin into site etc? Also which approach did you use to migrate? Did you use some official doco?

7

Unfortunately we have only testing environment with a user …Not exact similar to production ,In production we use AD for authentication .

Also we took inputs from below Community post and did the migration .

8

If I were you, I would roll back to sqlite

Then do a full test of the migration with full on actual set of users fully test it.

The article you references is just the migration part, it does not address the crucial part of testing.
Otherwise you just trying to keep alive a wounded beast

9

Thanks for the suggestion .We will follow accordingly and see if that helps us.We also thought of moving back but wanted to check here if any solutions were there .

1 Like