Docker Swarm + NFS Persistent Volumes

Grafana Installation
1

I’ve got the following compose:

version: '3'
services:
  grafana:
    image: grafana/grafana:latest
    ports:
      - '37000:3000'
    volumes:
      - grafana-storage:/grafana-storage
      - grafana-data:/grafana-data
      - grafana-provisioning:/grafana-provisioning
    environment:
      - GF_SECURITY_ADMIN_USER=${GRAFANA_USERNAME}
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
      - GF_PATHS_STORAGE="/grafana-provisioning"
#      - GF_PATHS_DATA="/grafana-data"
#      - GF_PATHS_CONFIG="/grafana-storage/grafana.ini"
    deploy:
      placement:
        constraints:
         - node.role==worker
   
volumes:    
  grafana-data:
    driver: local
    driver_opts:
      device: :/DockerDisks/grafana/storage/data
      o: addr=${NAS_HOST},rw,vers=4
      type: nfs      
  grafana-storage:
    driver: local
    driver_opts:
      device: :/DockerDisks/grafana/storage
      o: addr=${NAS_HOST},rw,vers=4
      type: nfs      
  grafana-provisioning:
    driver: local
    driver_opts:
      device: :/DockerDisks/grafana/provisioning
      o: addr=${NAS_HOST},rw,vers=4
      type: nfs

If I enable GF_PATHS_DATA or GF_PATHS_CONFIG line, I’m greeted with permission denied errors and failure to deploy.

Within the container itself, with those lines commented out.

ls -lah / shows

drwxr-xr-x    1 root     root        4.0K Oct 15 19:10 .
drwxr-xr-x    1 root     root        4.0K Oct 15 19:10 ..
-rwxr-xr-x    1 root     root           0 Oct 15 19:10 .dockerenv
drwxr-xr-x    1 root     root        4.0K Oct  8 10:20 bin
drwxr-xr-x    5 root     root         340 Oct 15 19:10 dev
drwxr-xr-x    1 root     root        4.0K Oct 15 19:10 etc
drwxrwxrwx    2 grafana  grafana     4.0K Oct 15 18:39 grafana-data
drwxrwxrwx    6 grafana  grafana     4.0K Oct 13 17:56 grafana-provisioning
drwxrwxrwx    3 grafana  grafana     4.0K Oct 15 18:37 grafana-storage
drwxr-xr-x    1 root     root        4.0K Oct  8 10:20 home
drwxr-xr-x    1 root     root        4.0K Oct  8 10:20 lib
drwxr-xr-x    5 root     root        4.0K May 29 14:21 media
drwxr-xr-x    2 root     root        4.0K May 29 14:21 mnt
drwxr-xr-x    2 root     root        4.0K May 29 14:21 opt
dr-xr-xr-x  208 root     root           0 Oct 15 19:10 proc
drwx------    2 root     root        4.0K May 29 14:21 root
drwxr-xr-x    1 root     root        4.0K Oct  8 10:20 run
-rwxr-xr-x    1 root     root        3.3K Oct  8 10:19 run.sh
drwxr-xr-x    2 root     root        4.0K May 29 14:21 sbin
drwxr-xr-x    2 root     root        4.0K May 29 14:21 srv
dr-xr-xr-x   12 root     root           0 Oct 15 19:10 sys
drwxrwxrwt    2 root     root        4.0K May 29 14:21 tmp
drwxr-xr-x    1 root     root        4.0K Oct  8 10:20 usr
drwxr-xr-x    1 root     root        4.0K Oct  8 10:20 var

from the NFS host, the relevant folder shows

drwxrwxrwx  4  472   472 4.0K Oct  1 13:55 grafana

All that says to me is that the permissions line up between NFS server, and container. Plus, the 777 permissions should let anything read/write into the NFS store.

What’s going on here that I’m missing?

2

I have found a similar issue on github: https://github.com/grafana/grafana/issues/24595

With GF_PATHS_DATA and GF_PATHS_CONFIG commented out, I can boot the container, log in as user grafana with a user/group id of 472:472 and read out of the relevant folders, and write to them as well.

I don’t know what the issue is here. Is there some sort of folder checking that is done prior to a full boot? How is everyone else handling a docker-swarm?

3

Final result:

version: '3'
services:
  grafana:
    image: grafana/grafana:latest
    ports:
      - '37000:3000'
    volumes:
      - grafana-storage:/etc/grafana
      - grafana-data:/var/lib/grafana
      - grafana-provisioning:/grafana-provisioning
    environment:
      - GF_SECURITY_ADMIN_USER=${GRAFANA_USERNAME}
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
      - GF_PATHS_STORAGE="/grafana-provisioning"
    deploy:
      placement:
        constraints:
         - node.role==worker
   
volumes:    
  grafana-data:
    driver: local
    driver_opts:
      device: :/DockerDisks/grafana/data
      o: addr=${NAS_HOST},rw,vers=4
      type: nfs      
  grafana-storage:
    driver: local
    driver_opts:
      device: :/DockerDisks/grafana/storage
      o: addr=${NAS_HOST},rw,vers=4
      type: nfs      
  grafana-provisioning:
    driver: local
    driver_opts:
      device: :/DockerDisks/grafana/provisioning
      o: addr=${NAS_HOST},rw,vers=4
      type: nfs

This works.

Setting the environment variables fails in my environment, but mapping the volume to the expected default locations works as expected.

I’m going to mark this solved, but it really isn’t, because the documentation seems misleading to me.