Migration 8.5.9 to 9.2.5 fails with dashboard migration permission issue

[niteshchauhan]

• What Grafana version and what operating system are you using?
  grafana 8.5.9 and OS centos 7
  Linux gec-co-graf01 3.10.0-1062.1.2.el7.x86_64 #1 SMP Mon Sep 30 14:19:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

• What are you trying to achieve?
  upgrade to version 9.2.5

• How are you trying to achieve it?
  we are using a grafana docker container with sqlite database
  we are trying to run another container with image 9.2.5 with same database

• What happened?
  migration fails with following logs

logger=migrator t=2022-12-16T11:52:11.641947998Z level=warn msg="alert linked to obsolete notification channel, ignoring" alert="glc-dsb-redis01 - Redis Uptime alert" uid=ZCyZgxfMz
logger=migrator t=2022-12-16T11:52:11.641955625Z level=warn msg="alert linked to obsolete notification channel, ignoring" alert="gdc-als-re03 - SWAP Usage alert" uid=5
logger=migrator t=2022-12-16T11:52:11.64196354Z level=warn msg="alert linked to obsolete notification channel, ignoring" alert="gdc-als-re03 - SWAP Usage alert" uid=8
logger=migrator t=2022-12-16T11:52:12.481690314Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations"
logger=migrator t=2022-12-16T11:52:12.713124885Z level=info msg="Executing migration" id="dashboard permissions"
logger=migrator t=2022-12-16T11:52:12.738219442Z level=error msg="Executing migration failed" id="dashboard permissions" error="failed to migrate permissions: failed to create permissions for role: UNIQUE constraint failed: permission.role_id, permission.action, permission.scope"
logger=migrator t=2022-12-16T11:52:12.738258674Z level=error msg="Exec failed" error="failed to migrate permissions: failed to create permissions for role: UNIQUE constraint failed: permission.role_id, permission.action, permission.scope" sql="code migration"
Failed to start grafana. error: migration failed (id = dashboard permissions): failed to migrate permissions: failed to create permissions for role: UNIQUE constraint failed: permission.role_id, permission.action, permission.scope
migration failed (id = dashboard permissions): failed to migrate permissions: failed to create permissions for role: UNIQUE constraint failed: permission.role_id, permission.action, permission.scope

• What did you expect to happen?
  succesful migration to 9.2.5

• Can you copy/paste the configuration(s) that you are having problems with?
  standard config with 8.5.9, Please let me know if want to know specific values

• Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
  UI couldn’t come up

• Did you follow any online instructions? If so, what is the URL?
  yes
  Role-based access control in Grafana 9.0

[antonio]

Welcome @niteshchauhan

Not sure but here’s a similar issue that might help you to overcome the issue. Note the comment with the SQL script.

Error 'Permissions needed: dashboards:read' when switching from 8.X to 9.X for anonymous access

Also, are you using provisioning to assing permissions?

[niteshchauhan]

Thanks for the information, before raising the issue, i saw this issue and tried this solution, but it doesn’t help. also as i see the that issue not resolved yet.
the above problem i stated is with a sqlite based grafana.
we today tried on a mysql based grafana too, and it also fails while migrating with following logs. this also is a 8.5.9 to 9.2.5 upgrade.

logger=migrator t=2022-12-20T08:28:06.64546023Z level=warn msg="alert linked to obsolete notification channel, ignoring" alert="tpc-co-elas03 - Disk Usage % alert" uid=5
logger=migrator t=2022-12-20T08:28:06.645480197Z level=warn msg="alert linked to obsolete notification channel, ignoring" alert="tpc-co-elas03 - Disk Usage % alert" uid=8
logger=migrator t=2022-12-20T08:28:06.645505942Z level=warn msg="alert linked to obsolete notification channel, ignoring" alert="TPC-MS-TAREDS03 - Memory Usage alert" uid=5
logger=migrator t=2022-12-20T08:28:06.645522684Z level=warn msg="alert linked to obsolete notification channel, ignoring" alert="TPC-MS-TAREDS03 - Memory Usage alert" uid=8
logger=migrator t=2022-12-20T08:28:06.64554253Z level=warn msg="alert linked to obsolete notification channel, ignoring" alert="tpc-appd-anco01 - Netstat alert" uid=5
logger=migrator t=2022-12-20T08:28:06.645556138Z level=warn msg="alert linked to obsolete notification channel, ignoring" alert="tpc-appd-anco01 - Netstat alert" uid=8
logger=migrator t=2022-12-20T08:28:15.496701782Z level=error msg="Executing migration failed" id="move dashboard alerts to unified alerting" error="Error 1062: Duplicate entry '1-ZpF-i_cVz' for key 'UQE_alert_rule_org_id_uid'"
logger=migrator t=2022-12-20T08:28:15.496811706Z level=error msg="Exec failed" error="Error 1062: Duplicate entry '1-ZpF-i_cVz' for key 'UQE_alert_rule_org_id_uid'" sql="code migration"

[niteshchauhan]

Hi, any updates on this issue. we tried to upgrade with legacy alerting and it got successfully upgraded.

[unified_alerting]
enabled = false

But we want the upgrade to work with v9 alerting, Please let us know if we can do something for this.

[antonio]

@niteshchauhan

I reviewed your case with my team and it seems you would need to address the error you first reported failed to create permissions for role: UNIQUE constraint failed: permission.role_id, permission.action, permission.scope"

Because the action names were renamed, this will cause errors due to custom roles that use previous action names. See this pull request to get more insight about RBAC action name changes.

With that in mind, you might need to update any scripts and provisioning files that are using the old action names.

If you get stuck, let us know and we will consider further steps.

[niteshchauhan]

Thanks for your reply.
i hope i understood what is required to try

• we are not using any scripts or provisioning files.
• 9.0.0-beta3 is suggested to use for migration, i used it, it failed with same error.

[antonio]

you are welcome @niteshchauhan

Do you know whether you’ve created any custom RBAC roles in the past?

I am going to throw some more ideas of what to look into buy they might not help, so take your time.

In your DB

• If you check the roles table, do you have any roles that are not prefixed by fixed , managed or basic ?

• Could you copy/paste in here your permission table please?

[niteshchauhan]

There were no custom roles created in the past, and i think with v8.5, RBAC was not there, and i dont see any options even now in grafana v8.5.9 with legacy alerting for creating roles.

• there are only managed roles

sqlite> select * from role ;
1|managed:users:5:permissions||1|1|managed_1_users_5|2022-11-18 09:10:37.172367591+00:00|2022-11-18 09:10:37.172367591+00:00|||0
2|managed:users:34:permissions||1|1|managed_1_users_34|2022-11-18 09:10:37.172367591+00:00|2022-11-18 09:10:37.172367591+00:00|||0
3|managed:users:4:permissions||1|1|managed_1_users_4|2022-11-18 09:10:37.172367591+00:00|2022-11-18 09:10:37.172367591+00:00|||0
4|managed:users:42:permissions||1|1|managed_1_users_42|2022-11-18 09:10:37.172367591+00:00|2022-11-18 09:10:37.172367591+00:00|||0
5|managed:users:54:permissions||1|1|managed_1_users_54|2022-11-18 09:10:37.172367591+00:00|2022-11-18 09:10:37.172367591+00:00|||0

• permissions table

sqlite> select * from permission ;
1|1|teams:read|teams:id:1|2022-11-18 09:10:37|2022-11-18 09:10:37
2|2|teams:read|teams:id:1|2022-11-18 09:10:37|2022-11-18 09:10:37
3|3|teams:read|teams:id:1|2022-11-18 09:10:37|2022-11-18 09:10:37
4|4|teams:read|teams:id:1|2022-11-18 09:10:37|2022-11-18 09:10:37
5|5|teams:read|teams:id:1|2022-11-18 09:10:37|2022-11-18 09:10:37

[antonio]

thank you for the info @niteshchauhan . We will take a look into it anyway, just in case.

Can I ask you to give us yet one more detail please? What steps have you taken to make the migration and in which order?

Also, is there any relationship between the containers you’ve mentioned?:

@niteshchauhan

we are trying to run another container with image 9.2.5 with same database

[niteshchauhan]

Happy 2023 Antonio,
for migration i did the following:

• grafana container is running with image 8.5.9 and use a sqlite database file which is mounted to the container.
• stop the container with 8.5.9 image.
• run another container with 9.2.5 image mounting the same volumes where configuration and database files are present.

[niteshchauhan]

Hello @antonio
Any progress on this, let me know if there any other solutions i can try to resolve this.
Thanks

[usman.ahmad]

Hi @niteshchauhan,

Not sure if you have already looked at this post where many users actively posted the process to migrate/update Grafana:

How to Move/Migrate Grafana to a newer version/system on a local/container environment

Hi, I want to migrate a grafana installation (v 5.3.4) to a new server. Is there a guide\docs or best practices to move grafana (without losing data,plugins, configurations, dashboard, datasource, …) from an old server to a newer? Old Server OS: Ubuntu 16.04 New Server OS: Debian 9.6.0 Thanks !

I hope this helps.

[niteshchauhan]

Hi @usman.ahmad
My problem is about migrating from legacy alerting to unified alerting. Please check above comments for errors faced.
The link you have sent is for migrating with the same config and version. it doesn’t help.
Thanks

[antonio]

@niteshchauhan Please bear with us while I get feedback from some of our engineers.

[antonio]

@niteshchauhan

it seems that Grafana 9.3 should contain a fix for this issue. You might want to try to upgrade to that version instead of 9.2.5.

Good luck

[niteshchauhan]

Hello @antonio
I’ve tried following which doesn’t helps.

• migrating from legacy alerting to unified alerting in v8.5.9
• migrating to v9.0.0, v9.2.5, v 9.3.0 with unified alerting.

Thanks

[yosiasz]

Copy your sqllite db elsewhere and query the alerts table wherein one of the columns has a value of 1-ZpF-i_cVz and see what is so odd/peculiar about that/those rows

[niteshchauhan]

Hello @yosiasz
This error was from our another env, which is running grafana 8.5.9 with mysql. in the comment i just gave an example that this is also failing to migrate.
for the sake solving this issue, i would still like to stick the error i have specified in problem statement, which is grafana 8.5.9 witih sqlite db.
i have pasted the content of tables here

Migration 8.5.9 to 9.2.5 fails with dashboard migration permission issue

There were no custom roles created in the past, and i think with v8.5, RBAC was not there, and i dont see any options even now in grafana v8.5.9 with legacy alerting for creating roles. there are only managed roles sqlite> select * from role ; 1|managed:users:5:permissions||1|1|managed_1_users_5|2022-11-18 09:10:37.172367591+00:00|2022-11-18 09:10:37.172367591+00:00|||0 2|managed:users:34:permissions||1|1|managed_1_users_34|2022-11-18 09:10:37.172367591+00:00|2022-11-18 09:10:37.172367591+00:…

[yosiasz]

@niteshchauhan

Duplicate entry '1-ZpF-i_cVz' for

There is this issue also with alerts

[niteshchauhan]

Please read the comment carefully, this issue is from some other env.
the problem in hand is having following logs

Failed to start grafana. error: migration failed (id = dashboard permissions): failed to migrate permissions: failed to create permissions for role: UNIQUE constraint failed: permission.role_id, permission.action, permission.scope
migration failed (id = dashboard permissions): failed to migrate permissions: failed to create permissions for role: UNIQUE constraint failed: permission.role_id, permission.action, permission.scope