Grafana Fails to run Migration during Upgrade

  • What Grafana version and what operating system are you using?

ubuntu 22.04, grafana 11.6.7

  • What are you trying to achieve?

upgrade grafana to version 11.6.8

  • How are you trying to achieve it?

via ubuntu standard package upgrade,

 sudo apt-get install grafana=11.6.8
  • What happened?

grafana fails to start, throwing an error about failing to run a migration. This failed migration actually crashes mysql [ percona-server-server 8.0.45-36-1.jammy ]. There is no indication in the changelog (https://github.com/grafana/grafana/blob/main/CHANGELOG.md#1168-2025-11-19), that there were any changes in 11.6.8

2026-04-06T13:34:00Z UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=0832afafe1217abd80e6981fb17855f1345ae8d3
Server Version: 8.0.45-36 Percona Server (GPL), Release '36', Revision '8fe4a72d'

Thread pointer: 0x7f936c013f30

40364 logger=migrator t=2026-04-06T09:33:59.97842203-04:00 level=debug msg="Skipping migration: Already executed" id="drop my_row_id and add primary key to file table i      f my_row_id exists (auto-generated mysql column)"
40365 logger=migrator t=2026-04-06T09:33:59.980921148-04:00 level=info msg="Executing migration" id="drop file_path unique index from file table if it exists (mysql)"
40366 logger=migrator t=2026-04-06T09:33:59.980949476-04:00 level=debug msg="Executing migration condition SQL" id="drop file_path unique index from file table if it ex      ists (mysql)" sql="SELECT 1 FROM `INFORMATION_SCHEMA`.`STATISTICS` WHERE `TABLE_SCHEMA` = DATABASE() AND `TABLE_NAME`=? AND `INDEX_NAME`=?" args="[file UQE_file_p      ath_hash]"
40367 logger=migrator t=2026-04-06T09:33:59.982636376-04:00 level=debug msg="Executing sql migration" id="drop file_path unique index from file table if it exists (mysq      l)" sql="ALTER TABLE file DROP INDEX UQE_file_path_hash"
40368 logger=migrator t=2026-04-06T09:34:00.105869443-04:00 level=error msg="Executing migration failed" id="drop file_path unique index from file table if it exists (m      ysql)" error="invalid connection" duration=124.943421ms
40369 logger=migrator t=2026-04-06T09:34:00.105945307-04:00 level=error msg="Exec failed" error="invalid connection" sql="ALTER TABLE file DROP INDEX UQE_file_path_hash      "
40370 logger=migrator t=2026-04-06T09:34:00.106106513-04:00 level=info msg="Unlocking database"
40371 logger=migrator t=2026-04-06T09:34:00.106195517-04:00 level=error msg="Failed to unlock database" error="invalid connection"
40372 logger=settings t=2026-04-06T09:34:00.851181369-04:00 level=info msg="Starting Grafana" version=11.6.8 commit=c3e34314f7e137bde0d27194f4f0933256681f1b branch=rele      ase-11.6.8 compiled=2026-04-06T09:34:00-04:00
  • What did you expect to happen?

Grafana successfully upgrade and start up properly

  • Can you copy/paste the configuration(s) that you are having problems with?
##################### Grafana Configuration Example #####################
#
# Everything has defaults so you only need to uncomment things you want to
# change

# possible values : production, development
; app_mode = production

# instance name, defaults to HOSTNAME environment variable value or hostname if HOSTNAME var is empty
; instance_name = ${HOSTNAME}

#################################### Paths ####################################
[paths]
# Path to where grafana can store temp files, sessions, and the sqlite3 db (if that is used)
#
;data = /var/lib/grafana
#
# Directory where grafana can store logs
#
;logs = /var/log/grafana
#
# Directory where grafana will automatically scan and look for plugins
#
;plugins = /var/lib/grafana/plugins

#
#################################### Server ####################################
[server]
domain = hostname.internal
root_url = %(protocol)s://%(domain)s:/grafana-engine

# Protocol (http, https, socket)
;protocol = http

# The ip address to bind to, empty will bind to all interfaces
;http_addr =

# The http port  to use
;http_port = 3000

# The public facing domain name used to access grafana from a browser
;domain = localhost

# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
;enforce_domain = false

# The full public facing url you use in browser, used for redirects and emails
# If you use reverse proxy and sub path specify full url (with sub path)
;root_url = http://localhost:3000

# Log web requests
;router_logging = false

# the path relative working path
;static_root_path = public

# enable gzip
;enable_gzip = false

# https certs & key file
;cert_file =
;cert_key =

# Unix socket path
;socket =

#################################### Database ####################################
[database]
# You can configure the database connection by specifying type, host, name, user and password
# as seperate properties or as on string using the url propertie.

# Either "mysql", "postgres" or "sqlite3", it's your choice
;type = mysql
;host = 127.0.0.1:3306
;name = grafana
;user = root
# If the password contains # or ; you have to wrap it with trippel quotes. Ex """#password;"""
;password =

# Use either URL or the previous fields to configure the database
# Example: mysql://user:secret@host:port/database
url = mysql://redacted_username:redacted_password@hostname.internal:3306/grafana

# For "postgres" only, either "disable", "require" or "verify-full"
;ssl_mode = disable

# For "sqlite3" only, path relative to data_path setting
;path = grafana.db

# Max conn setting default is 0 (mean not set)
;max_idle_conn =
;max_open_conn =


#################################### Session ####################################
[session]
# Either "memory", "file", "redis", "mysql", "postgres", default is "file"
;provider = file

# Provider config options
# memory: not have any config yet
# file: session dir path, is relative to grafana data_path
# redis: config like redis server e.g. `addr=127.0.0.1:6379,pool_size=100,db=grafana`
# mysql: go-sql-driver/mysql dsn config string, e.g. `user:password@tcp(127.0.0.1:3306)/database_name`
# postgres: user=a password=b host=localhost port=5432 dbname=c sslmode=disable
;provider_config = sessions

# Session cookie name
;cookie_name = grafana_sess

# If you use session in https only, default is false
;cookie_secure = false

# Session life time, default is 86400
;session_life_time = 86400

#################################### Data proxy ###########################
[dataproxy]

# This enables data proxy logging, default is false
;logging = false


#################################### Analytics ####################################
[analytics]
# Server reporting, sends usage counters to stats.grafana.org every 24 hours.
# No ip addresses are being tracked, only simple counters to track
# running instances, dashboard and error counts. It is very helpful to us.
# Change this option to false to disable reporting.
;reporting_enabled = true

# Set to false to disable all checks to https://grafana.net
# for new vesions (grafana itself and plugins), check is used
# in some UI views to notify that grafana or plugin update exists
# This option does not cause any auto updates, nor send any information
# only a GET request to http://grafana.com to get latest versions
;check_for_updates = true

# Google Analytics universal tracking code, only enabled if you specify an id here
;google_analytics_ua_id =

#################################### Security ####################################
[security]
# default admin user, created on startup
admin_user = grafana_admin

# default admin password, can be changed before first start of grafana,  or in profile settings
admin_password = redacted

# used for signing
;secret_key = redacted

# Auto-login remember days
;login_remember_days = 7
;cookie_username = grafana_user
;cookie_remember_name = grafana_remember

# disable gravatar profile images
;disable_gravatar = false

# data source proxy whitelist (ip_or_domain:port separated by spaces)
;data_source_proxy_whitelist =

[snapshots]
# snapshot sharing options
;external_enabled = true
;external_snapshot_url = https://snapshots-origin.raintank.io
;external_snapshot_name = Publish to snapshot.raintank.io

# remove expired snapshot
;snapshot_remove_expired = true

# remove snapshots after 90 days
;snapshot_TTL_days = 90

#################################### Users ####################################
[users]
# disable user signup / registration
allow_sign_up = false

# Allow non admin users to create organizations
;allow_org_create = true

# Set to true to automatically assign new users to the default organization (id 1)
;auto_assign_org = true

# Default role new users will be automatically assigned (if disabled above is set to true)
;auto_assign_org_role = Viewer

# Background text for the user field on the login page
;login_hint = email or username

# Default UI theme ("dark" or "light")
;default_theme = dark

[auth]
# Set to true to disable (hide) the login form, useful if you use OAuth, defaults to false
;disable_login_form = false

# Set to true to disable the signout link in the side menu. useful if you use auth.proxy, defaults to false
;disable_signout_menu = false

#################################### Anonymous Auth ##########################
[auth.anonymous]
# enable anonymous access
enabled = false

# specify organization name that should be used for unauthenticated users
;org_name = Main Org.

# specify role for unauthenticated users
;org_role = Viewer

#################################### Basic Auth ##########################
[auth.basic]
enabled = true

#################################### Auth LDAP ##########################
[auth.ldap]
enabled = true
config_file = /etc/grafana/ldap.toml
allow_sign_up = true


#################################### Logging ##########################
[log]
# Either "console", "file", "syslog". Default is console and  file
# Use space to separate multiple modes, e.g. "console file"
;mode = console file

# Either "trace", "debug", "info", "warn", "error", "critical", default is "info"
;level = info

# optional settings to set different levels for specific loggers. Ex filters = sqlstore:debug
;filters =


# For "console" mode only
[log.console]
;level =

# log line format, valid options are text, console and json
;format = console

# For "file" mode only
[log.file]
;level =

# log line format, valid options are text, console and json
;format = text

# This enables automated log rotate(switch of following options), default is true
;log_rotate = true

# Max line number of single file, default is 1000000
;max_lines = 1000000

# Max size shift of single file, default is 28 means 1 << 28, 256MB
;max_size_shift = 28

# Segment log daily, default is true
;daily_rotate = true

# Expired days of log file(delete after max days), default is 7
;max_days = 7

[log.syslog]
;level =

# log line format, valid options are text, console and json
;format = text

# Syslog network type and address. This can be udp, tcp, or unix. If left blank, the default unix endpoints will be used.
;network =
;address =

# Syslog facility. user, daemon and local0 through local7 are valid.
;facility =

# Syslog tag. By default, the process' argv[0] is used.
;tag =


#################################### AMQP Event Publisher ##########################
[event_publisher]
;enabled = false
;rabbitmq_url = amqp://localhost/
;exchange = grafana_events

;#################################### Dashboard JSON files ##########################
[dashboards.json]
;enabled = false
;path = /var/lib/grafana/dashboards

#################################### Alerting ############################
[alerting]
# Disable alerting engine & UI features
enabled = false
# Makes it possible to turn off alert rule execution but alerting UI is visible
;execute_alerts = true

#################################### Internal Grafana Metrics ##########################
# Metrics available at HTTP API Url /api/metrics
[metrics]
# Disable / Enable internal metrics
;enabled           = true

# Publish interval
;interval_seconds  = 10

# Send internal metrics to Graphite
[metrics.graphite]
# Enable by setting the address setting (ex localhost:2003)
;address =
;prefix = prod.grafana.%(instance_name)s.

#################################### Grafana.com integration  ##########################
# Url used to to import dashboards directly from Grafana.com
[grafana_com]
;url = https://grafana.com

#################################### External image storage ##########################
[external_image_storage]
# Used for uploading images to public servers so they can be included in slack/email messages.
# you can choose between (s3, webdav)
;provider =

[external_image_storage.s3]
;bucket_url =
;access_key =
;secret_key =

[external_image_storage.webdav]
;url =
;public_url =
;username =
;password =

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

  • Did you follow any online instructions? If so, what is the URL?

hello @jgrammenosagilitypr , before migration must take grafana.db backup , please read attached documnet for grafana migration.


and restart the grafana service
also check grafana.log file.
thanks

I read through the linked instructions. They recommend backing up the database and my configuration. Then running the upgrade. I have done so with the same results.

grafana.log snippet

40364 logger=migrator t=2026-04-06T09:33:59.97842203-04:00 level=debug msg=“Skipping migration: Already executed” id=“drop my_row_id and add primary key to file table i f my_row_id exists (auto-generated mysql column)”
40365 logger=migrator t=2026-04-06T09:33:59.980921148-04:00 level=info msg=“Executing migration” id=“drop file_path unique index from file table if it exists (mysql)”
40366 logger=migrator t=2026-04-06T09:33:59.980949476-04:00 level=debug msg=“Executing migration condition SQL” id=“drop file_path unique index from file table if it ex ists (mysql)” sql=“SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME=? AND INDEX_NAME=?” args=“[file UQE_file_p ath_hash]”
40367 logger=migrator t=2026-04-06T09:33:59.982636376-04:00 level=debug msg=“Executing sql migration” id=“drop file_path unique index from file table if it exists (mysq l)” sql=“ALTER TABLE file DROP INDEX UQE_file_path_hash”
40368 logger=migrator t=2026-04-06T09:34:00.105869443-04:00 level=error msg=“Executing migration failed” id=“drop file_path unique index from file table if it exists (m ysql)” error=“invalid connection” duration=124.943421ms
40369 logger=migrator t=2026-04-06T09:34:00.105945307-04:00 level=error msg=“Exec failed” error=“invalid connection” sql="ALTER TABLE file DROP INDEX UQE_file_path_hash "
40370 logger=migrator t=2026-04-06T09:34:00.106106513-04:00 level=info msg=“Unlocking database”
40371 logger=migrator t=2026-04-06T09:34:00.106195517-04:00 level=error msg=“Failed to unlock database” error=“invalid connection”
40372 logger=settings t=2026-04-06T09:34:00.851181369-04:00 level=info msg=“Starting Grafana” version=11.6.8 commit=c3e34314f7e137bde0d27194f4f0933256681f1b branch=rele ase-11.6.8 compiled=2026-04-06T09:34:00-04:00

grafana still fails to run the migrations in part because the query it is attempting to run causes mysql to hard crash

What user are you using to connect to myself and does that user have enough permission to make schema and indices changes

Seems like a permission issue

Yes, the user I am using to connect to mysql does have enough permissions as far as I can tell. Other migrations seems to run fine.

mysql> show grants for ‘redacted_username’@‘redacted_ip_filter’;
±--------------------------------------------------------------+
| Grants for redacted_username@redacted_ip_filter                                 |
±--------------------------------------------------------------+
| GRANT USAGE ON *.* TO redacted_username@redacted_ip_filter                  |
| GRANT ALL PRIVILEGES ON grafana.* TO redacted_username@redacted_ip_filter |
±--------------------------------------------------------------+

the user has full privileges on the grafana database and all of its tables. and the IP filter matches the source ip.
The migration with id drop file_path unique index from file table if it exists (mysql) and sql ALTER TABLE file DROP INDEX UQE_file_path_hash is the problematic one. why would dropping an index cause mysql to crash?

1 Like

So this part of your xonfig is confusing

# Either “mysql”, “postgres” or “sqlite3”, it’s your choice

;type = mysql

;host = 127.0.0.1:3306

;name = grafana

;user = root

Mysql is commented out.

Because I am using the url style of providing the database connection details, visible in the configs a few lines down from there

1 Like

So the next possible issue might be timeouts

It could be the table from which the index is being dropped from might be huge or some other constraint such as network and timeout might be causing it.

So Grafana does error while trying to run the migration:

the connection has become invalid.
the reason for the invalid connection: mysql crashed
2026-04-06T13:34:00Z UTC - mysqld got signal 11 ;

I do not know at this time why trying to run that particular query ALTER TABLE file DROP INDEX UQE_file_path_hash, causes a mysql crash, but it is not the behaviour I would expect