So I have good news and bad news. The good news is that several engineers on the team have encountered situations like this before, and there is a way to fix it. The bad news is that the fix is very manual; we don’t currently have any script that will fix this for you, although perhaps now we should develop one for the community.
Here are some notes from the team to help you. Please let me know if this info helps you get unblocked. Together we’ll get this fixed.
description of problem
the tl;dr from one senior engineer: “you can trick Grafana into recreating an index for a table by removing rows from the
first identify which indexes are missing, then you have to look in the
migration_log table to see where it says it was created and delete that row.
The team has seen this when Grafana gets stopped during an upgrade, like the container gets killed exactly during an in-process upgrade. Or MySQL fell over during the upgrade. Also: had you been fiddling with MySQL directly before this happened?
for your scenario, it may be enough to just get the list of missing indexes, and then delete the corresponding rows from the
migration_log table, and on startup grafana will make them
Grafana executes database migrations when it starts up. It has a list of statements that it runs if they are not in the
One way to do this is to go into the
migration_log and delete the rows that create the indexes. Then restart Grafana and it will run those statements again. The tricky part is that he says some and not all. If an index already exists then the migrations will not work.
You could run a query like this:
where migration_id like '%index%';
- See attached screenshot for what this query will return.
There might be a faster or more clever way to do this, but I hope that this method works for you. If possible, I’d back up the DB and test all this in a dev environment before you attempt any fixes.
Let me know if this helps!