How to backup Grafana in Kubernetes system

Grafana Configuration
1

Hi

I am new to Grafana and also K8S. I just deployed Grafana using Helm into my Rancher K8S system and I want to know how to backup it.

I tried to follow this document but still confused because I am not sure where $WORKING_DIR or customs.ini are.

Please help.

Thank you very much.
KL

2

It depends. Let’s assume you are using volume for Grafana DB, so just snapshot that volume (it’s advance k8 topic):

1 Like
3

Thank you for your answer. I also want to do that as it seems like the easiest way to backup.
However, I have got this issue ‘/var/lib/grafana/plugins’: permission denied’ when I enable persistent volume using Helm as instructed on this link
Deploy Grafana using Helm Charts | Grafana documentation

4

It depends on your personal preference - there is no the best solution. I hate K8S volumes, so I use SaaS DB (e.g. AWS RDS Aurora) with automated daily snapshots with 30d retention period + manual snapshot before each major upgrade, I have restore exercise once per year. This is my “the easiest way”.

5
  1. is your grafana backend sqlite?
  2. how many dashboards do you have?
  3. do you have any other fancy stuff going on in your grafana other than dashboards?
6

@yosiasz

  1. I deployed it through helm without editing anything. So, I think it is on sqllite.
  2. Since I just started with it, I have developed 2 dashboards.
  3. No, my plan was to use it for visualizing time series data from machines in the factory and alert when some anomalies happen such as values exceeding thresholds.
1 Like
7

highly recommend you also look at the provisioning of dashboards route

8

Hello
Is there a way to pause the Grafana DB (SQLite) during the snapshot?
(indeed, if modifications occur during snapshot, then DB may not be consistent, and after restoring the Volume, the DB may be corrupt , right?)

9

Yes, stop Grafana pod(s).

10

If I stop grafana pod, then grafana service will be cut during the backup
If I do not want to cut grafana service during backup, is there a way to export sqlite db using CLI?

1 Like
11

which cli?

12

Correct. So how do you want to have a consistent backup, when you are using file-based SQLite DB?
Serious deployments don’t use SQLite, but proper DB (MySQL, PostgreSQL). Users use SQLite because it’s the default DB, so they don’t need to lift a finger to configure DB. Then they complain, e.g. Search results for '"Database locked" order:latest' - Grafana Labs Community Forums
You have what you configured. In your case, it’s easy deployment but terrible backups.

1 Like
13

This is why you also need to seriously look at provisioning