Root or not-root for gathering log files

I looking for a good way to give the grafana-agent minimal permissions but still be able to gather logs from docker containers. This is related to the documentation given on the docker integration.

The most straight forward integration would be to just use

        - job_name: docker
          static_configs:
            - targets: [localhost]
              labels:
                job: docker
                instance: {{ hostname }}
                __path__: /var/lib/docker/containers/*/*-json.log

but the files and directories are all owned by root. And giving the grafana-agent full root access does not sound like the smartest idea. Maybe using ACLs could be an alternative:

setfacl -R -m group:grafana-agent:rX /var/lib/docker/containers

But for this to be inherited to new containers it would require some owner changes on some docker directories - and I didn’t have the guts to try this on a live system.

It’s a shame the docker files are not root:docker owned and at least readable to the docker group. Given the level of access of members of that group - it would make sense.

For better or worse it seems like access /var/run/docker.sock can provide access to docker - including access to logs.

        - job_name: integrations/docker
          docker_sd_configs:
            - host: unix:///var/run/docker.sock
              refresh_interval: 5s
          relabel_configs:
            - action: replace
              replacement: integrations/docker
              source_labels:
                - __meta_docker_container_id
              target_label: job
            - source_labels:
                - __meta_docker_container_name
              regex: '/(.*)'
              target_label: container
            - source_labels:
                - __meta_docker_container_log_stream
              target_label: stream

I suspect the access to the unix socket is why it requires the grafana-agent become part of the docker group to work. Then again this is giving full access to docker and by that extend to root. Something to be aware of.

Where I am still a bit wondering is whether using the cadvisor is a requirement for docker_sd_configs or not. It’s also a shame all this re-labelling has to be done explicitly/manually in the config.

I am just wondering what people use for gather logs. Especially when there is the goal to give up the least possible permissions.

The available docs on this and the current situation is not ideal.

Here are my following findings:

It is possible to switch the docker process from root:root to root:docker by changing the service file. Unfortunately docker still creates the files as root:root and has no configuration option (that I know of) to change the target group of the file ownerships. This would have been my preferred solution as it would allow for a read-only access.

Setting the ACLs could be an option when running through an orchestrator. We are using Nomad and could in theory adjust the ACLs after every container start. This feels quite hacky though. I gave that a miss.

What I ended up doing is to add the grafana agent to the docker group and use the docker unix socket to get access to the logs. This is a bad compromise. But still better than running the grafana agent as root.

What is not obvious from the docs - you can use the discovery docker_sd_configs without activating the cadvisor integration.

So the following snippet works for us:

        - job_name: integrations/docker
          docker_sd_configs:
            - host: unix:///var/run/docker.sock
              refresh_interval: 5s
          relabel_configs:
            - action: replace
              replacement: integrations/docker
              source_labels:
                - __meta_docker_container_id
              target_label: job
            - source_labels:
                - __meta_docker_container_name
              regex: '/(.*)'
              target_label: container
            - source_labels:
                - __meta_docker_container_log_stream
              target_label: stream
            - source_labels:
                - __meta_docker_container_label_service
              target_label: service

Another thing that I have’t seen/found/overlooked in the documentation is that container labels get exposed via __meta_docker_container_label_ prefix.

I wish there was a way to look at the labels that are coming in. This makes the relabelling part very awkward for inputs that are found via discovery.

As side note: It’s a little insane that everyone has to configre these relabelings instead of integrating this into the grafana agent itself.