Docker Swarm + NFS Persistent Volumes

I’ve got the following compose:

version: '3'
services:
  grafana:
    image: grafana/grafana:latest
    ports:
      - '37000:3000'
    volumes:
      - grafana-storage:/grafana-storage
      - grafana-data:/grafana-data
      - grafana-provisioning:/grafana-provisioning
    environment:
      - GF_SECURITY_ADMIN_USER=${GRAFANA_USERNAME}
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
      - GF_PATHS_STORAGE="/grafana-provisioning"
#      - GF_PATHS_DATA="/grafana-data"
#      - GF_PATHS_CONFIG="/grafana-storage/grafana.ini"
    deploy:
      placement:
        constraints:
         - node.role==worker
   
volumes:    
  grafana-data:
    driver: local
    driver_opts:
      device: :/DockerDisks/grafana/storage/data
      o: addr=${NAS_HOST},rw,vers=4
      type: nfs      
  grafana-storage:
    driver: local
    driver_opts:
      device: :/DockerDisks/grafana/storage
      o: addr=${NAS_HOST},rw,vers=4
      type: nfs      
  grafana-provisioning:
    driver: local
    driver_opts:
      device: :/DockerDisks/grafana/provisioning
      o: addr=${NAS_HOST},rw,vers=4
      type: nfs       

If I enable GF_PATHS_DATA or GF_PATHS_CONFIG line, I’m greeted with permission denied errors and failure to deploy.

Within the container itself, with those lines commented out.

ls -lah / shows

drwxr-xr-x    1 root     root        4.0K Oct 15 19:10 .
drwxr-xr-x    1 root     root        4.0K Oct 15 19:10 ..
-rwxr-xr-x    1 root     root           0 Oct 15 19:10 .dockerenv
drwxr-xr-x    1 root     root        4.0K Oct  8 10:20 bin
drwxr-xr-x    5 root     root         340 Oct 15 19:10 dev
drwxr-xr-x    1 root     root        4.0K Oct 15 19:10 etc
drwxrwxrwx    2 grafana  grafana     4.0K Oct 15 18:39 grafana-data
drwxrwxrwx    6 grafana  grafana     4.0K Oct 13 17:56 grafana-provisioning
drwxrwxrwx    3 grafana  grafana     4.0K Oct 15 18:37 grafana-storage
drwxr-xr-x    1 root     root        4.0K Oct  8 10:20 home
drwxr-xr-x    1 root     root        4.0K Oct  8 10:20 lib
drwxr-xr-x    5 root     root        4.0K May 29 14:21 media
drwxr-xr-x    2 root     root        4.0K May 29 14:21 mnt
drwxr-xr-x    2 root     root        4.0K May 29 14:21 opt
dr-xr-xr-x  208 root     root           0 Oct 15 19:10 proc
drwx------    2 root     root        4.0K May 29 14:21 root
drwxr-xr-x    1 root     root        4.0K Oct  8 10:20 run
-rwxr-xr-x    1 root     root        3.3K Oct  8 10:19 run.sh
drwxr-xr-x    2 root     root        4.0K May 29 14:21 sbin
drwxr-xr-x    2 root     root        4.0K May 29 14:21 srv
dr-xr-xr-x   12 root     root           0 Oct 15 19:10 sys
drwxrwxrwt    2 root     root        4.0K May 29 14:21 tmp
drwxr-xr-x    1 root     root        4.0K Oct  8 10:20 usr
drwxr-xr-x    1 root     root        4.0K Oct  8 10:20 var

from the NFS host, the relevant folder shows

drwxrwxrwx  4  472   472 4.0K Oct  1 13:55 grafana

All that says to me is that the permissions line up between NFS server, and container. Plus, the 777 permissions should let anything read/write into the NFS store.

What’s going on here that I’m missing?