Setup MySQL highly available cluster as a database for grafana

Grafana Installation
1

Hello,
We are looking to setup grafana for high availability. I read through the document related to it here:
http://docs.grafana.org/tutorials/ha_setup/

In existing setup we have 3 grafana-server machines connecting to 3 machine master-slave mysql cluster.

We are planning to have a HA MySql cluster setup with multiple master nodes distributed across couple of different geographical locations. There will be more grafana server machines and all of them will talk to MySql cluster via a load balancer.

My question is: do we need to have any change in grafana schema to enable it to work with multi master mysql setup? Are there any limitations (for e.g. grafana to be aware of a column that is created as primary key or such) in terms of using multi master mysql setup?

Thanks for your help and suggestions.

2

Not sure, you will have to consult MySQL documentation on how multi master setup works and what implications that have.

Actually we might run multi master MySQL for GrafanClouds Hosted Grafana service, will ask Ops team.

3

Hello Nikhil,

I am in the same boat as yours. Spent some time in connecting grafana to multi master mariadb galero cluster but no success. I saw it created multiople records instead of 1 in tables with no primary key. It would be great if you can help me if you have figured out the solution to this.

4

Sure Komal I’ll share what i learn with multi-master tryout.
So did you see same row repeated multiple times in a table? Any specific tables? Could you please share some more details about what you tried?

5

I am truing to push sessions into mariadb galero instead of files
SInce mariadb galero do not support Myisam storage engine, should I use innodb storage engine ?

When I am using innodb storage engine, I get deadlock error:
EROR[05-25|20:51:10] Request error logger=context userId=0 orgId=0 uname= error=“session(release): Error 1213: Deadlock found when trying to get lock; try restarting transaction” stack="/usr/local/go/src/runtime/panic.go:489

MyISam storage engine didn’t work as mariadb galero doesmn’t support it hence replication didn’t happen.

6

With MariaDB innodb , do you run master-slave or multi master?

7

I have a similar setup. I am using 3 instances of grafana behind a k8s service. I have a mysql-galera cluster with 3 nodes. I am using this to store the dashboards, and grafana sessions. I have created the session table with type InnoDB. The replication is going through well. I have 2 dashboards and some alerts configured on them. The issue i am having is of session creation. Grafana instances creating too many sessions behind the scenes. Ex In a days time around 25k sessions. Is this a regular thing? Is there a configuration that i can use so that the http calls don’t create a new session every time? seems like these calls are going directly from grafana server and not originating from UI. Also i noted that the cleanup service only cleans up tmp files not the sessions in mysql. Over a period of time should we clean the sessions by ourselves? Following are the info logs…

2017-05-27T07:22:02.825419378Z t=2017-05-27T07:22:02+0000 lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=GET path=/metrics status=404 remote_addr=10.99.57.0 time_ms=12ns size=7763
2017-05-27T07:22:02.825819566Z t=2017-05-27T07:22:02+0000 lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=GET path=/metrics status=404 remote_addr=10.99.100.0 time_ms=13ns size=7763
2017-05-27T07:22:02.971217777Z t=2017-05-27T07:22:02+0000 lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=GET path=/metrics status=404 remote_addr=10.99.9.117

8

With MariaDB innodb ,I run multi-master. Also I see the example in the documentation suggests to use MyISAM storage engine, is this required ?