Grafana Docker and data persistence

tam481 · July 15, 2020, 9:47pm

Hello all,
I am trying to get my Grafana docker container to persist data. I’ve configured the container to run as root to rule out any user account related issues. I’m not sure what I’m doing wrong but I can’t get the data to persist.

I can remove panels from the default Home dashboard. If I create a custom dashboard everything seems to be OK but as soon as I log out and re-login (to the default admin account) it is as if I’m logging in for the first time.

Here’s my docker-compose file

  grafana:
    image: grafana/grafana:latest
    user: root
    restart: unless-stopped
    container_name: grafana
    volumes:
       - $PWD/data/grafana:/var/lib/grafana
       - $PWD/conf/grafana:/etc/grafana
       - $PWD/logs/grafana:/var/log/grafana
       - $PWD/data/grafana/provisioning:/etc/grafana/provisioning
    environment:
      - "GF_INSTALL_PLUGINS=grafana-clock-panel,grafana-simple-json-datasource,grafana-piechart-panel"
    networks:
      - backend

What am I doing wrong? Initially, the container complained about not finding directories (dashboard, datasources and notifiers) within the provisioning directory which I expected it to create under /etc/grafana so to overcome that I added the last binding and manually created these directories under the provisioning directory.

Thank you in advance

kanekennedy · July 23, 2020, 8:45am

Have you made the data directories first? Are the permissions on them correct? Personally i hate using $PWD and always set a full path. I like using /docker_data as it makes it obvious to anyone after me what it is.

Try this…

version: '3'

services:
  grafana:
    image: grafana/grafana
    container_name: grafana
    depends_on:
      - influxdb
    ports:
      - 3000:3000
    volumes:
      - /docker_data/grafana_data:/var/lib/grafana
      - /docker_data/certs:/certs
    env_file:
      - /docker_data/grafana_conf/config.monitoring 
    environment:
      - HTTP_USER="{{ grafana_user }}"
      - HTTP_PASS="{{ grafana_passwd }}"
      - INFLUXDB_HOST=influxdb
      - INFLUXDB_PORT=8086
      - INFLUXDB_NAME="{{ db_name }}"
      - INFLUXDB_USER="{{ influxdb_user }}"
      - INFLUXDB_PASS="{{ influxdb_passwd }}"
    restart: always
   
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    volumes:
      - /docker_data/prometheus_conf:/etc/prometheus/
      - /docker_data/prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
    ports:
      - 9090:9090
    links:
      - grafana
      - influxdb
    restart: always 
  
  influxdb:
    image: docker.io/influxdb
    container_name: influxdb
    ports:
      - "8086:8086"
    volumes:
      - /docker_data/influxdb_data:/var/lib/influxdb
    env_file:
      - '/docker_data/influxdb_conf/env.influxdb' 
    restart: always

That should give you a stack.

I use Ansible for all the lifting -

docker-install.yml

--- 
    - 
      connection: localhost
      hosts: localhost
      tasks: 
        - 
          args: 
            chdir: /home/
          command: "curl -fsSL get.docker.com -o get-docker.sh"
          name: "Download docker script"
        - 
          name: "run docker script"
          script: /home/get-docker.sh
        - 
          name: "start Docker"
          service: 
            name: docker
            state: started
        - 
          apt: 
            name: python-pip
            state: present
          name: "install pip"
          register: results
        - 
          debug: 
            var: results
        - 
          command: "pip install docker-compose"
          name: "install docker-compose"
        - 
          name: "include vars file"
          include_vars: docker_install.var.yml
        - 
          name: "create docker data folders"
          file:
            path: "{{ item }}"
            state: directory
            mode: 0777
          with_items:
              - /docker_data/portainer_data
              - /docker_data/influxdb_data
              - /docker_data/grafana_data
              - /docker_data/chronograf_data
              - /docker_data/prometheus_data
        -      
          name: "create docker config folders"
          file:
            path: "{{ item }}"
            state: directory
            mode: 0775
          with_items:
              - /docker_data/influxdb_conf
              - /docker_data/grafana_conf
              - /docker_data/prometheus_conf
        - 
          name: "create telegraf.conf from template values"
          template: 
            dest: /docker_data/telegraf_conf/telegraf.conf
            src: telegraf.conf.jk2
        - 
          name: "create telegraf.conf outputs from template values"
          template: 
            dest: /docker_data/telegraf_conf/telegraf.d/telegraf.outputs.conf
            src: telegraf.outputs.conf.j2
        - 
          name: "create telegraf.conf inputs from template values"
          template: 
            dest: /docker_data/telegraf_conf/telegraf.d/telegraf.inputs.conf
            src: telegraf.inputs.jk2        
        - 
          file: 
            mode: 420
            path: /docker_data/telegraf_conf/telegraf.conf
          name: "change file permission"
        - 
          name: "create grafanacfg.output file with values"
          template: 
            dest: /docker_data/grafana_conf/grafanacfg.output
            src: grafanacfg.input.jk2
        - 
          name: "create influx env file"
          template: 
            dest: /docker_data/influxdb_conf/env.influxdb
            src: env.influxdb.j2
        - 
          name: "create grafana config file"
          template: 
            dest: /docker_data/grafana_conf/config.monitoring
            src: config.monitoring.j2
        - 
          name: "create docker-compose.yml from template"
          template: 
            dest: /docker_data/docker-compose.yml
            src: docker-compose.yml.j2
        - 
          name: "create docker-compose .env from template"
          template: 
            dest: /docker_data/.env
            src: docker-compose.env.j2    
        - 
          args: 
            chdir: /docker_data/
          command: "docker-compose build"
          name: "docker build"
        - 
          args: 
            chdir: /docker_data/
          command: "docker-compose up -d"
          name: "docker up"
        - 
          name: "Wait 300 seconds for port 3000 to become open on the host, don't start checking for 10 seconds"
          wait_for: 
            delay: 10
            port: 3000
        - 
          name: "Set Grafana admin password"
          command: docker exec -t grafana bash -c  'grafana-cli --homepath /usr/share/grafana admin reset-admin-password "{{ grafana_passwd }}"'
        - 
          name: "check if grafana can be accessed"
          command: "curl -s http://grafana.staged-by-discourse.com/api/org -u {{ grafana_user }}:{{ grafana_passwd }}"
        - 
          name: "create datasoure in grafana"
          command: "curl -L --header 'Content-Type: application/json' --header 'Accept: application/json' -d@//docker_data/grafana_conf/grafanacfg.output http://grafana.staged-by-discourse.com/api/datasources -u {{ grafana_user }}:{{ grafana_passwd }}"
        - 
          name: "create TelegrafHost dashboard in grafana using jsonfile"
          command: "curl -L --header 'Content-Type: application/json' --header 'Accept: application/json' -d@/home/Configfiles/dashboards/TelegrafHostMetrics.json  http://grafana.staged-by-discourse.com/api/dashboards/db -u {{ grafana_user }}:{{ grafana_passwd }}"
        - 
          name: "create DockerMetrics dashboard in grafana using jsonfile"
          command: "curl --header 'Content-Type: application/json' --header 'Accept: application/json' -d@/home/Configfiles/dashboards/DockerMetricsperContainer.json  http://grafana.staged-by-discourse.com/api/dashboards/db -u {{ grafana_user }}:{{ grafana_passwd }}"
        - 
          name: "create JVM dashboard in grafana using jsonfile"
          command: "curl --header 'Content-Type: application/json' --header 'Accept: application/json' -d@/home/Configfiles/dashboards/jvm-metrics-jolokia-2_rev1.json  http://grafana.staged-by-discourse.com/api/dashboards/db -u {{ grafana_user }}:{{ grafana_passwd }}"

docker-install.var.yml (example this wont work!)

grafana_user: !vault |
        $ANSIBLE_VAULT;1.1;AES256
    
grafana_passwd: !vault |
        $ANSIBLE_VAULT;1.1;AES256

influxdb_admin_user: !vault |
        $ANSIBLE_VAULT;1.1;AES256

influxdb_admin_passwd: !vault |
        $ANSIBLE_VAULT;1.1;AES256

influxdb_user: !vault |
        $ANSIBLE_VAULT;1.1;AES256

influxdb_passwd: !vault |
        $ANSIBLE_VAULT;1.1;AES256

I suggest you use ansible-vault to not save passwords in plain text.

This should give you a good start, there are some files you will need to create, they are mainly your current config files saved as .jk2 or j2 (where I forgot the k). The dashborads I auto deploy can be removed or you can go to the Grafana page for them and download the yml.

tam481 · July 23, 2020, 4:37pm

Thank you very much for your help.
Since posting, I noticed that I am able to create and save dashboards etc. What I cannot do is reset the admin password (it keeps prompting me to change it every time" I also cannot seem to be able to edit the main “home” dashboard.

I thought it had something to do with Grafana being behind Traefik so I tested a standalone container with port 3000 exposed on http and found that it did the same thing so I know for sure it’s not Traefik.

Any ideas?

alex440 · September 18, 2021, 12:25pm

@tam481
I have the same problem. How did you solve it?

javi5 · May 8, 2022, 10:57am

To do so:

Stop and remove the grafana docker container
Remove the grafana volumes
Adjust below environment setting, for grafana container:

      GF_SECURITY_ADMIN_USER

Re-create again the grafana docker container

brian22ca · March 17, 2024, 9:59pm

Hey! For me, I had Grafana listening to Prometheus in a docker container. Although I created a volume for Grafana, I did not create a volume for Prometheus. Since Prometheus’ data was reset every time, Grafana also lost all the data as a result. Simply adding a volume to persist Prometheus data fixed the issue for me, as detailed here: persistence - How to persist data in Prometheus running in a Docker container? - Stack Overflow

Topic		Replies	Views
Docker container delete all my dashboards	4	5905	February 15, 2020
New Docker Install with persistent storage, Permission problem Configuration docker , permissions	20	111506	December 5, 2024
Add user in Grafana Docker Configuration docker , grafana-roles	2	3305	February 6, 2023
Persistent volume for user accounts - Docker Configuration docker	3	10792	November 7, 2018
Got Grafana up and running without persistent volume .... now what? Grafana docker	1	2142	October 1, 2019

Grafana Docker and data persistence

Related topics