Promtail - rewriting output (and using internal labels)

Best practice with Loki is to create as few labels as possible and to use the power of stream queries. To this end, it suggests that even a small number of labels combined with a small number of values can cause problems.

Therefore when scraping syslog it would seem sensible to not create labels for all syslog internal fields.

This leaves the problem of how to retain that data, as it would be lost if simply discarded by not setting relabel configs for it.

An option might be to adjust the message content to include this data. For example, including the facility or severity in the message content itself rather than as a label.

The problem seems to be that the internal labels do not appear to be available in either replace or templates.

Example with input syslog message of “hello world”

scrape_configs:
  - job_name: syslog
    syslog:
      listen_address: 127.0.0.1:1514
      idle_timeout: 60s
      label_structured_data: yes
      labels:
        job: "syslog"
        host: myhostname
    pipeline_stages:
      - replace:
          expression: "(?P<content>.*)"
          replace: '[{{ .__syslog_message_severity }}] {{ .Value }}'

This results in a message output of:
[<no value>] hello world

Am I going about rewriting the message content wrongly here, or is there a problem with internal labels being used on replace or template stages?

1 Like