Hello all -
I have Grafana set up for visualization of monitoring data from Prometheus. I am fetching the data from there as my source and then using mysql to hold the dashboards and other persistent data. In the past two months, I have lost the Grafana database twice. The Grafana container will be in a constant restart loop with a container log error saying it can’t find the Grafana database. I use the Mysql workbench to access mysql db’s. Sure enough if I connect to the mysql server within the container, there is no Grafana db. So, not sure if I have something wrong? If I should be using some other type of persistent data mechanism other than mysql? If my implementation of mysql using azure disk is bad\wrong\etc?
I then have to restore the backup of the DB but lose metrics from the time it went missing. Any help would be greatly appreciated! Thank you!
Why are you losing metrics? The Grafana database only saves settings like dashboards and users - your metrics should be in Prometheus?
This doesn’t sound like it is a problem with Grafana if you accidentally lose your database. Sounds like a configuration problem with your docker volumes.
I’m new to this environment and trying to figure things out on the fly. On closer inspection, what it looks like preceded this was an azure level event with the linux vm instance that hosts the mysql db. I have a mysql db just to dump dashboard and persistence data. Everything in the cluster came back up once the vm recovered. But the grafana dashboards did not, that mysql db is not resilient to a restart event apparently. Again, just trying to piece this all together. I understand that grafana db only stores those dashboards and user settings, and looking at how to best architect that component to be resilient (not necessarily HA, just resilient to events and able to recover without intervention). So yes, something is wrong in the mysql configuration perhaps. It’s currently installed on an azure disk mounted to the linux vm instance. Is it possible to not use mysql and just write those to flat file / blob storage?
Thanks again for the help!
It is possible to use the default embedded Sqlite database which is a file and that can be stored in blob storage I think. MySQL is needed if you have a lot of users making changes to dashboards, I would guess that more than 80% of Grafana users use Sqlite.
The marketplace offering is slightly out of date (we are working on adding it to the Grafana release cycle so that it gets updated automatically) but it might be a good starting point for you: https://azuremarketplace.microsoft.com/en-in/marketplace/apps/grafana-labs.grafana_oss