Configuration property __path__ could by dynamic?

Grafana Loki
1

HI all,

I got a file configuration like below…
but there is no way to have this property (path) dynamic?
'Cause if I recreate my contaier or rebuil from my docker-compose I got another ID…
I would like somethig related to the service name which is constant.

Thanks Alen


....
- job_name: exporter
  pipeline_stages:
  - docker: {}
  static_configs:
  - targets:
      - localhost
    labels:
      job: exporter
      __path__: /var/lib/docker/containers/c22c0933c266275467ecf28a7ed259d8b9e9572131b55*/*-json.log
....
2

Yeah, you can of course make it dynamic, you are already doing it with a wildcard in the path, just add another one:

- job_name: exporter
  pipeline_stages:
  - docker: {}
  static_configs:
  - targets:
      - localhost
    labels:
      job: exporter
      __path__: /var/lib/docker/containers/*/*.log
3

Thanks Tony… but in this way I catch all my FOLDER , all my docker app!
AS you can see I use just a pattern…
containers/c22c0933c266275467ecf28a7ed259d8b9e9572131b55*/*-json.log

I need this dynamic… but related to my APP, maybe it’s not possible…
c22c0933c266275467ecf28a7ed259d8b9e9572131b55

It’s not a solution…
Thank anyway.
ALEN
: (

4

It is a solution, you just need to filter it. When you configure json driver, you can provide a label like so:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "1g",
    "max-file": "10",
    "labels": "logs_i_care_about"
  }
}

And then you filter it in your promtail configuration:

scrape_configs:
- job_name: docker-logs
  static_configs:
  - targets:
      - localhost
    labels:
      job: docker-logs
      __path__: /var/lib/docker/containers/*/*log

  pipeline_stages:
  - json:
      expressions:
        attrs:
  - json:
      expressions:
        tag:
      source: attrs
  - labels:
      tag:
  - match:
      selector: '{tag="logs_i_care_about"}'
      stages:
      ### Do something ###
  - match: # Drop logs that we don't care.
      selector: '{tag=!"logs_i_care_about"}'
      action: drop
      drop_counter_reason: log_i_do_not_care_about

You’ll want to verify the promtail configuration yourself, i just typed it and it’s not tested.

5

Also, just to be clear, your original configuration will break as soon as you refresh your container, because the ID will have changed. So it’s already not a working configuration.

6

Hi Tony,
and sorry for the late in my response…

but it’s no clear to me…
below the snippet of my 2 docker app that I would like be related to the app name and not to the ID of container…

server:
  http_listen_port: 9080
  grpc_listen_port: 0

clients:
  - url: http://loki:3100/loki/api/v1/push
  
positions:
  filename: /var/log/positions.yaml

scrape_configs:
- job_name: sqlserver-exporter
  pipeline_stages:
  - docker: {}
  static_configs:
  - targets:
      - localhost
    labels:
      job: sqlserver-exporter
      __path__: /var/lib/docker/containers/abea8a3d90685609103c5cf984cfde035f*/*-json.log 


- job_name: sqlserver-exporter-webapp
  pipeline_stages:
  - docker: {}
  static_configs:
  - targets:
      - localhost
    labels:
      job: sqlserver-exporter-webapp
      __path__: /var/lib/docker/containers/457fca631b5bbaeb8e1bd28bc15e96fd98e*/*-json.log

It’s not clear to me where I must insert this snippet…

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "1g",
    "max-file": "10",
    "labels": "logs_i_care_about"
  }
}

Thanks ALEN

7

Here is what I recommend you to do:

  1. Configure your Docker container to include a tag when outputting logs. This is the log-driver configuration you see above. If you are not sure how to do this, look up Configure logging drivers | Docker Documentation, depending on what you do with Docker (K8S or ECS or some other platform) it may be different how you inject that configuration.

  2. Change your promtail configuration to include wildcard in the path (/var/lib/docker/containers/*/*.log), and then use a match block to filter only the containers you want by tag (injected by step 1).

1 Like