Add container name to promtail docker logs

[kai]

I run successfully a centralized loki logging for several docker servers with multiple images running on them. When I now look via grafana into the logs and needs to filter for one virtual container output I have no hint for the docker container name.

current promtail config is partly this one:

# which logs to read/scrape
scrape_configs:
  - job_name: docker-logs
    pipeline_stages:
    - docker: {}
    static_configs:
    - targets: # tells promtail to look for the logs on the current machine/host
        - localhost
      labels: # labels with which all the following logs should be labelled
        job: container
        host: docker    # label-1
        __path__: /var/lib/docker/containers/*/*-json.log
        # __path__ denotes the path of the actual log file whose logs will be scraped
        # and shipped to loki indexed with the above labels

Now in the log view I only see the hostname where all docker images are running under.

In addition I would like to extract the NAMES property of that docker, like diun here.

[b0b]

Hi @kai ,

I got curious and Googled this. This is not something I have tested myself but I have struggled with Promtail configurations in the past

Looking at this, there seems to be a good example that is called job_name: containers in the first version of docker-config.yml.

Hope that helps.

Without using service discovery, I think you have to take care of all the labeling config yourself. Don’t think there is any mechanism for automatic Docker metadata discovery…

[kai]

That seems to be a way, sadly the syntax has changed since that post.
So I will have to play around with that instead of just C&P to a solution. I’ll reply again if I get any further. Thanks so far.

[kai]

I just found out that simply using the loki logging plugin for docker already adds the tags for container and composer I needed. So now it is with all the filter info I needed.

[dorinand]

Hi @kai ,

recently, I was solving similar issue like you - use Promtail on Docker swarm with docker metadata. At first I tried loki logging plugin but I soon ran into problems. In case loki is unreachable, logging plugin is trying to sent logs to loki and depends on your configuration, two things can occur:

1. Promtail will drop logs after X retries
2. Promtail will keep trying forever

In both cases, this cause that I could not see logs in docker logs during time period when plugin was trying to send logs and even not able to kill the container because of this. This is known issue of loki plugin:

The driver keeps all logs in memory and will drop log entries if Loki is not reachable and if the quantity of max_retries has been exceeded. To avoid the dropping of log entries, setting max_retries to zero allows unlimited retries; the drive will continue trying forever until Loki is again reachable. Trying forever may have undesired consequences, because the Docker daemon will wait for the Loki driver to process all logs of a container, until the container is removed. Thus, the Docker daemon might wait forever if the container is stuck.

In my opinion, best solution is to use Promtail with Docker service discover. It is able to get metadata from docker like container name, id, network, labels, etc… To use it you have to use replabel config, example from doc:

scrape_configs:
  - job_name: flog_scrape 
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
        filters:
          - name: name
            values: [flog] 
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'

Be careful with this configuration, label container, even with one app, can contain a lot of uniq values because of docker hash name like:

• nginx_nginx.1.0zwo879s92fk38o07a65uzolc - replicated deployment
• minio_minio.wy6c6pk1arod6vbqeem8iyosj.t9mdy2hgnv2upqwmrh0fo2c9p - global deployment

This is not recommended according to Grafana Loki label best pracitce. I fix this with regexp relabel config which filter name of the container, in above example nginx_nginx or minio_minio , and replica number (if exists):

scrape_configs:
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)\.[0-9]\..*'
        target_label: 'name'
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)\.[0-9a-z]*\..*'
        target_label: 'name'
      - source_labels: ['__meta_docker_container_name']
        regex: '/.*\.([0-9]{1,2})\..*'
        target_label: 'replica'