Unable to get folder name with promtail regexp

Grafana Loki
1

this is my config file for the promtail

server:
  disable: true
client:
  url: http://loki-gateway.loki/loki/api/v1/push
positions:
  filename: /tmp/positions.yaml
scrape_configs:
  - job_name: grabbers_log
    static_configs:
      - targets:
          - localhost
        labels:
          job: grabbers_log
          pod: ${POD_NAME}
          node_name: ${NODE_NAME}
          namespace: ${NAMESPACE}
          __path__: /var/logs/scrapyd/logs/grabbers/**/*.log
    # this will make the folder name as an attribute key
    # in loki
    relabel_configs:
      - source_labels: [__path__]
        regex: '/var/logs/scrapyd/logs/grabbers/(.*)/.*' # '.*/grabbers/([^/]+)/.*\.log'
        target_label: grabber

everything works, but I can’t get additional label added for the record based on the last folder name, i.e */grabbers/ERA 2024/* I need a label named grabber=ERA 2024 in loki, but instead I get grabber=** in grafana

xx
xx1414×303 32 KB

Any ideas what am I doing wrong?

2

You are using __path__ as source, so /var/logs/scrapyd/logs/grabbers/**/*.log is processed and you get ** in grabbers from it

filename should be used as source to get what you need.

See also this topic

1 Like
3

so I tried to use the following

relabel_configs:
                - source_labels: [__filename__]
                  regex: '.*/grabbers/([^/]+)/.*\.log'
                  target_label: grabber
                  replacement: '$1'
                  action: replace

and now the label grabbers is not there at all.

4

ok the working solution:

configMap:
    enabled: true
    files:
      grabbers-log:
        promtail.yaml: |-
          server:
            disable: true
            log_level: debug
          client:
            url: http://loki-gateway.loki/loki/api/v1/push
          positions:
            filename: /tmp/positions.yaml
          scrape_configs:
            - job_name: grabbers_log
              static_configs:
                - targets:
                    - localhost
                  labels:
                    job: grabbers_log
                    pod: ${POD_NAME}
                    node_name: ${NODE_NAME}
                    namespace: ${NAMESPACE}
                    __path__: /var/logs/scrapyd/logs/grabbers/**/*.log
              # this will make the folder name as an attribute key
              # in loki so we can query by grabber name
              # {job="grabbers_log", grabber="ERA 2024"}
              pipeline_stages:
                - match:
                    selector: '{job="grabbers_log"}'
                    stages:
                      - regex:
                          source: filename
                          expression: '.*\/grabbers\/(?P<grabber>.*)\/.*'
                      - labels:
                          grabber: