Two grafana servers with single sqlite3 databse on NFS


#1

Is it safe to deploy Grafana on two different servers and point the grafana.db file to the shared NFS share?


#2

I don’t think that’s a good idea if both grafana servers are active at the same time since it could corrupt the database.
If you want to have a HA setup I strongly suggest you use mysql/postgre.


#3

According to sqlite documentation it should not corrupt the database: https://sqlite.org/faq.html#q5
"Multiple processes can have the same database open at the same time. Multiple processes can be doing a SELECT at the same time. But only one process can be making changes to the database at any moment in time, however."

I just want to understand if grafana does anything which won’t be safe (say non transactional writes, e.t.c.)


#4

I didn’t know that.

Looking at the docs for sqlite it seems all operations will end up in a transaction regardless if grafana starts one.
“No changes can be made to the database except within a transaction. Any command that changes the database (basically, any SQL command other than SELECT) will automatically start a transaction if one is not already in effect. Automatically started transactions are committed when the last query finishes.” https://sqlite.org/lang_transaction.html

That said… We are not experts in sqlite or NFS. We have only tested sqlite accessed from one instance.


#5

NOT SAFE - A SQLite DB should not be shared from an NFS share, corruption/ data integrity issues will ensue. SQLite is excellent, NFS is the issue. You might have better luck with WebDAV/SabreDAV with full lock support enabled, but I have not tried it.

Reference - search for NFS:
https://www.sqlite.org/howtocorrupt.html
https://sqlite.org/faq.html


#6

Thank you for clarifying.


#7

According to these links sqlite is using fcntl. In modern Linux Kernels the fcntl on NFS should be safe:
http://nfs.sourceforge.net/#faq_d10

After two weeks of testing it looks good so far


#8

Update: it is not safe, the database got corrupted after some time returning “cannot start a transaction within a transaction” error