Unable to Provision Grafana (Docker) | /usr/share/grafana is not readable

[dvdlbrecht]

Hi,

I’m trying to implement provisioning as described here for my Docker container so it can automatically connect to the datasource and bring up a few default dashboards.

I’m on Raspberry Pi OS Lite (Buster 10) and Docker version 20.10.6.

When I create a new container, I see in the Docker logs:

$ docker logs grafana -n 1000
GF_PATHS_CONFIG='/etc/grafana/grafana.ini' is not readable.
GF_PATHS_DATA='/var/lib/grafana' is not writable.
GF_PATHS_HOME='/usr/share/grafana' is not readable.
You may have issues with file permissions, more information here: http://docs.grafana.org/installation/docker/#migrate-to-v51-or-later

So, Grafana isn’t able to pull my provisioning files out of /usr/share/grafana/conf/provisioning/, and I am left with an un-provisioned container.

Here’s how I’m creating the new container. I have tried a variety of --user settings when creating the container as recommended by the link in the log message above, and some other questions I’ve found relating to this issue, to no avail.

docker run -d --restart always --name grafana -p 3000:3000 --user 472:0 -v /home/pi/grafana-provisioning:/usr/share/grafana/conf/provisioning grafana/grafana:8.2.2

As you can see, I am mapping the local directory /home/pi/grafana-provisioning/ to the provisioning folder inside the container.

The permissions and ownership on the local grafana-provisioning directory are set very liberally:

$ ls -lsht
total 296K
4.0K drwxrwxrwx  4 pi pi 4.0K Oct 29 12:18 grafana-provisioning

As are the contents inside…

$ ls -lsht grafana-provisioning/
total 8.0K
4.0K drwxrwxrwx 2 pi pi 4.0K Oct 29 12:08 datasources
4.0K drwxrwxrwx 2 pi pi 4.0K Oct 29 12:08 dashboards

I have tried various UID and GID settings (removing the container and recreating it between each attempt). All the options I’ve tried result in the same permissions log entries.

Looking inside the docker container, the /usr/share/grafana directory is owned by root:

$ ls -lsht
total 28K
     4 drwxr-xr-x    1 root     root        4.0K Oct 21  2021 grafana

… As are the files and folders inside it:

$ ls -lsht grafana/
total 68K
     4 -rw-r--r--    1 root     root           5 Oct 21  2021 VERSION
     4 drwxr-xr-x    2 root     root        4.0K Oct 21  2021 bin
     4 drwxr-xr-x    3 root     root        4.0K Oct 21  2021 conf
     4 drwxr-xr-x    3 root     root        4.0K Oct 21  2021 plugins-bundled
     4 drwxr-xr-x    2 root     root        4.0K Oct 21  2021 scripts
     4 drwxr-xr-x   16 root     root        4.0K Oct 21  2021 public
    36 -rw-r--r--    1 root     root       33.7K Oct 21  2021 LICENSE
     4 -rw-r--r--    1 root     root         108 Oct 21  2021 NOTICE.md
     4 -rw-r--r--    1 root     root        2.8K Oct 21  2021 README.md

Since root owns the grafana folder inside the container, I’ve tried setting the docker run --user flag to root:root, 0:0, and 472:0:, and none of these combinations seem to work.

The default UID and GID inside the container, when the --user flag is not set in docker run, is:

$ echo "`id -u`:`id -g`"
472:0

What am I doing wrong?

edit: I am going to make a custom Dockerfile just so I can chown the /usr/share/grafana folder while building the image. I have a good feeling this will work, but it just doesn’t seem like this should be necessary in order to get support for provisioning Grafana when deployed via Docker.

[dvdlbrecht]

I ended up figuring this out after digging deeper.

It turns out that the message in the docker logs about potential file permissions errors was a total distraction. A few more lines down from that line, I noticed the docker log messages:

msg="Config overridden from command line" logger=settings arg="default.paths.data=/var/lib/grafana"
msg="Config overridden from command line" logger=settings arg="default.paths.logs=/var/log/grafana"
msg="Config overridden from command line" logger=settings arg="default.paths.plugins=/var/lib/grafana/plugins"
msg="Config overridden from command line" logger=settings arg="default.paths.provisioning=/etc/grafana/provisioning"
msg="Config overridden from command line" logger=settings arg="default.log.mode=console"
msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana"
msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana"
msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"

More specifically, I found the default.paths.provisioning=/etc/grafana/provisioning which gave me exactly what I needed.

I whipped together a simple Dockerfile to map my build context to /etc/grafana/provisioning, and I use docker build to create a custom image just to map my provisioning files properly.

Here’s a quick rundown as I’m sure this will be helpful for someone in the future. Create a new folder in your home folder (or wherever you want, really), with the following sub-structure. Let’s say my new folder is simply called docker . Here are the contents of docker:

.
├── build_grafana.sh
├── Dockerfile
└── provisioning
    ├── dashboards
    │   ├── dashboards.yaml
    │   ├── individual_ct_data.json
    │   └── system.json
    └── datasources
        └── influx.yaml

3 directories, 6 files

Here are the contents of Dockerfile, which is at docker/Dockerfile

# syntax=docker/dockerfile:1

FROM grafana/grafana:8.2.3

ADD ./provisioning /etc/grafana/provisioning

build_grafana.sh is just a script that contains the following commands, which you must run while working in the docker folder since the context of the current dir (.) is passed to docker.

docker build . --tag grafana_v8.2.3

docker run -d --restart always --name grafana -p 3000:3000 grafana_v8.2.3:latest

The .json and .yaml files are my unique dashboards and datasource configs - the features of which are well-described in the Grafana official documentation.