Promtail extract token from __path__ with relabel

[wkwan]

HI all,

I have logs aggregated at /applogs/hostname/app.log. I’d like to have logs labelled with hostname and app. Since I may have 10 to 20 hostnames and a dozen of apps, I set _ _ path _ _ to /applogs/**/*.log

I have relabel setup as below, I get “**” as label hostname and “*” as label logname. What’s the best way to handle path with wildcards?

scrape_configs:
- job_name: applogs
  static_configs:
  - targets:
      - localhost
    labels:
      test: basic8
      __path__: /applogs/**/*.log
  relabel_configs:
    - source_labels: ["__path__"]
      regex: /applogs/([^/]+)/([^/]+)\.log
      replacement: $1
      target_label: hostname
      action: replace
    - source_labels: ["__path__"]
      regex: /applogs/([^/]+)/([^/]+)\.log
      replacement: $2
      target_label: logname
      action: replace

[tonyswumac]

Use regex + labels, something like (not tested):

pipeline_stages:
  - regex:
      expression: '^\/applogs\/(?P<hostname>[^\/]+)\/(?P<logname>[^\.]+)\.log'
      source: __path__
  - labels:
      hostname:
      logname:

[wkwan]

yeah… it is not getting the hostname and logname. There’re only the ‘filename’ and ‘test’ labels.

scrape_configs:
- job_name: applogs
  static_configs:
  - targets:
      - localhost
    labels:
      test: test12
      __path__: /applogs/*/*.log
  pipeline_stages:
    - regex:
        expression: '^\/applogs\/(?P<hostname>[^\/]+)\/(?P<logname>[^\.]+)\.log'
        source: __path__
    - labels:
        hostname:
        logname:

[tonyswumac]

Then use filename as the source of regex:

pipeline_stages:
    - regex:
        expression: '^\/applogs\/(?P<hostname>[^\/]+)\/(?P<logname>[^\.]+)\.log'
        source: filename
    - labels:
        hostname:
        logname:

[wkwan]

Thanks Tony
That works great. Got the lablels