Promtail: different pipeline stages on different log files

Grafana Loki
1

Hi,

This is my yaml file for promtail:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    pipeline_stages:
      # Written by Gemini
      - regex:
        expression: '(?P<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d+)\+\d{2}:\d{2}) (?P<host>\S+) (?P<severity>\S+),(?P<message>.+)'
        timestamp:
          type: timestamp
        host:
          type: string
        severity:
          type: string
        message:
          type: string
    static_configs:
    - targets:
        - localhost
      labels:
        job: varlogs
        __path__: /var/log/{messages,kern}.log

  - job_name: docker
    pipeline_stages:
      - docker:
    static_configs:
    - targets:
        - localhost
      labels:
        job: docker
        __path__: /var/log/docker.log

What I’m trying to do is run different pipeline stages on different log files as they have different formats. Unfortunately Promtail doesn’t like it and responds like this:

promtail-1  | level=info ts=2024-11-09T09:28:17.044338977Z caller=promtail.go:133 msg="Reloading configuration file" md5sum=2f477bea549d6246df9274f9e6e9eaa3
promtail-1  | level=info ts=2024-11-09T09:28:17.044338977Z caller=promtail.go:133 msg="Reloading configuration file" md5sum=2f477bea549d6246df9274f9e6e9eaa3
promtail-1  | level=error ts=2024-11-09T09:28:17.045242906Z caller=main.go:169 msg="error creating promtail" error="failed to make file target manager: pipeline stage must contain only one key"
promtail-1  | level=error ts=2024-11-09T09:28:17.045242906Z caller=main.go:169 msg="error creating promtail" error="failed to make file target manager: pipeline stage must contain only one key"
promtail-1  | level=info ts=2024-11-09T09:28:17.044338977Z caller=promtail.go:133 msg="Reloading configuration file" md5sum=2f477bea549d6246df9274f9e6e9eaa3
promtail-1  | level=error ts=2024-11-09T09:28:17.045242906Z caller=main.go:169 msg="error creating promtail" error="failed to make file target manager: pipeline stage must contain only one key"
promtail-1  | level=info ts=2024-11-09T09:28:17.044338977Z caller=promtail.go:133 msg="Reloading configuration file" md5sum=2f477bea549d6246df9274f9e6e9eaa3
promtail-1  | level=info ts=2024-11-09T09:28:17.044338977Z caller=promtail.go:133 msg="Reloading configuration file" md5sum=2f477bea549d6246df9274f9e6e9eaa3
promtail-1  | level=error ts=2024-11-09T09:28:17.045242906Z caller=main.go:169 msg="error creating promtail" error="failed to make file target manager: pipeline stage must contain only one key"
promtail-1  | level=info ts=2024-11-09T09:28:17.044338977Z caller=promtail.go:133 msg="Reloading configuration file" md5sum=2f477bea549d6246df9274f9e6e9eaa3
promtail-1  | level=error ts=2024-11-09T09:28:17.045242906Z caller=main.go:169 msg="error creating promtail" error="failed to make file target manager: pipeline stage must contain only one key"
promtail-1  | level=error ts=2024-11-09T09:28:17.045242906Z caller=main.go:169 msg="error creating promtail" error="failed to make file target manager: pipeline stage must contain only one key"
promtail-1  | level=info ts=2024-11-09T09:28:17.044338977Z caller=promtail.go:133 msg="Reloading configuration file" md5sum=2f477bea549d6246df9274f9e6e9eaa3
promtail-1  | level=error ts=2024-11-09T09:28:17.045242906Z caller=main.go:169 msg="error creating promtail" error="failed to make file target manager: pipeline stage must contain only one key"
promtail-1 exited with code 1
promtail-1 exited with code 1
promtail-1 exited with code 1
promtail-1 exited with code 1
promtail-1 exited with code 1
promtail-1 exited with code 1
promtail-1 exited with code 1

I can’t understand where the problem is here. Can anyone help?

Thanks in advance! :slight_smile:

2

Never mind people, I figured it out. It was a YAML indentation problem. Correct config looks like this:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  pipeline_stages:
    # Written by Gemini
    - regex:
        expression: '(?P<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d+\+\d{2}:\d{2}) (?P<host>\S+)(?P<severity>\S+) (?P<message>.+)'
        timestamp:
          type: timestamp
        host:
          type: string
        severity:
          type: string
        message:
          type: string
  static_configs:
  - targets:
    - localhost
    labels:
      job: varlogs
      __path__: /var/log/{messages,kern}.log

- job_name: docker
  pipeline_stages:
    - docker:
  static_configs:
  - targets:
    - localhost
    labels:
      job: docker
      __path__: /var/log/docker.log