Grafana 10.2.0 upgrade failure using mysql database

Grafana Installation
1

Hi,

I just upgrade grafana to 10.2.0 with mysql as database.
My grafana in Centos 7
I don;t have any issue in Ubuntu or Kubernetes, but Centos 7.

Below the log in the migration:

Oct 26 12:53:30 monitoring02 grafana: logger=settings t=2023-10-26T12:53:30.464549501+07:00 level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
Oct 26 12:53:30 monitoring02 grafana: logger=settings t=2023-10-26T12:53:30.464742164+07:00 level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
Oct 26 12:53:30 monitoring02 grafana: logger=settings t=2023-10-26T12:53:30.464766338+07:00 level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
Oct 26 12:53:30 monitoring02 grafana: logger=settings t=2023-10-26T12:53:30.464779899+07:00 level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
Oct 26 12:53:30 monitoring02 grafana: logger=settings t=2023-10-26T12:53:30.464791009+07:00 level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
Oct 26 12:53:30 monitoring02 grafana: logger=settings t=2023-10-26T12:53:30.464802086+07:00 level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
Oct 26 12:53:30 monitoring02 grafana: logger=settings t=2023-10-26T12:53:30.464820071+07:00 level=info msg=Target target=[all]
Oct 26 12:53:30 monitoring02 grafana: logger=settings t=2023-10-26T12:53:30.464869116+07:00 level=info msg="Path Home" path=/usr/share/grafana
Oct 26 12:53:30 monitoring02 grafana: logger=settings t=2023-10-26T12:53:30.46488246+07:00 level=info msg="Path Data" path=/var/lib/grafana
Oct 26 12:53:30 monitoring02 grafana: logger=settings t=2023-10-26T12:53:30.464893514+07:00 level=info msg="Path Logs" path=/var/log/grafana
Oct 26 12:53:30 monitoring02 grafana: logger=settings t=2023-10-26T12:53:30.464908835+07:00 level=info msg="Path Plugins" path=/var/lib/grafana/plugins
Oct 26 12:53:30 monitoring02 grafana: logger=settings t=2023-10-26T12:53:30.465015914+07:00 level=info msg="Path Provisioning" path=/etc/grafana/provisioning
Oct 26 12:53:30 monitoring02 grafana: logger=settings t=2023-10-26T12:53:30.465037051+07:00 level=info msg="App mode production"
Oct 26 12:53:30 monitoring02 grafana: logger=sqlstore t=2023-10-26T12:53:30.468758507+07:00 level=info msg="Connecting to DB" dbtype=mysql
Oct 26 12:53:30 monitoring02 grafana: logger=migrator t=2023-10-26T12:53:30.47788942+07:00 level=info msg="Starting DB migrations"
Oct 26 12:53:30 monitoring02 grafana: logger=migrator t=2023-10-26T12:53:30.498018871+07:00 level=info msg="Executing migration" id="add unique index signing_key.key_id"
Oct 26 12:53:30 monitoring02 grafana: logger=migrator t=2023-10-26T12:53:30.503698308+07:00 level=error msg="Executing migration failed" id="add unique index signing_key.key_id" error="Error 1071 (42000): Specified key was too long; max key length is 767 bytes" duration=5.657264ms
Oct 26 12:53:30 monitoring02 grafana: logger=migrator t=2023-10-26T12:53:30.503783865+07:00 level=error msg="Exec failed" error="Error 1071 (42000): Specified key was too long; max key length is 767 bytes" sql="CREATE UNIQUE INDEX `UQE_signing_key_key_id` ON `signing_key` (`key_id`);"
Oct 26 12:53:30 monitoring02 grafana: Error: ✗ migration failed (id = add unique index signing_key.key_id): Error 1071 (42000): Specified key was too long; max key length is 767 bytes
Oct 26 12:53:30 monitoring02 systemd: grafana-server.service: main process exited, code=exited, status=1/FAILURE

Please help to make it running. I can do mysql query for this one.

Regards,
Fadjar

2

Hello @fadjar340

What is the data type of column . (key_id) in table signing_key?

Where is the mysql installed? On same host as grafana?

What version of mysql is it

3

Hi @yosiasz ,

  1. The data tyoe column .(key_id) in table signing_key is varchar
  2. The Mysql/MariaDB id on another machine, not in the same host as Grafana
  3. The version is mysql Ver 15.1 Distrib 5.5.68-MariaDB
4

Varchar of what length

5

varchar(255)

6

change that to varchar(max) and rerun migration

7

There’s an error about varchar(max)

SQL Error [1064] [42000]: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'max) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL' at line 1
  You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'max) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL' at line 1
  You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'max) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL' at line 1
8

please post the sql syntax for which you are getting an error for? :wink:

9

I just made the varchar(255) to varchar(15000), because more than 15000 I have an error to change to TEXT or BLOB. Using 15000 can be done, but still can’t migrate.

ALTER TABLE grafana.signing_key MODIFY COLUMN key_id varchar(15000) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL;

10

the same error still happens or something else?

11

Yes, still the same as my first posting.

key was too long; max key length is 767 bytes

12

let’s try this. spin up a fresh 10.2.0 and point it to a totally different brand new database and see if it spins up successfully and then see what happens to key_id Then work our way back, reverse engineer it

13
  1. I’ve made new database called grafana1
  2. Change the database to that in grafana.ini
  3. Start upgrade to 10.2.0
  4. The error still same about the key

Now, I revert back to previous version 10.1.5 and the database

14

so we are not looking for an upgrade. we are looking for a brand new fresh install. but maybe 10.2.0 is problematic?

15

Perhaps…
This problem occurred since I have MariaDB as database and on another machine.
Since using sqlite is no problem.
I’m afraid it will problem also if using Kubernetes with the same configuration.

1 Like
16

Do you have some data in that table?

17

Nope,
the signing_key table was empty, in the old database or in the new database, as previous attempt.

1 Like
18

Another option might be to change the migration script to skip that table or submit a bug?

19

I’ll submit bug…

1 Like
20

I have the same issue. Is there a bug report I can see?