Promtail replacement string verified with --inspect, but original string sent to Grafana

Grafana Loki
1

Hello Community,

I have a log file with a lot of entries, one entry per line, 14 fields per entry, fields separated by semicolons. I want to “prettify” some of the entries before they are sent to Loki and then on to Grafana. Here is the relevant part of the log file including the entry where I want the “replace” stage to replace some XML-like tags with dashes:

26.12.23. 13:01:58 ;26.12.23. 13:00:35 ;ON   ;LogOnly ;1170 POSN failure                    ;secondary sensor failed Gps1    ;secondary sensor failed Gps1    ;5025111736321_G19_O171    ;CAUTION  ;                                ;CATEGORY B;3016;847;7
26.12.23. 13:02:05 ;26.12.23. 13:00:32 ;OFF  ;Valid   ;10549 HCS unavailable                 ;No gyro input                   ;**<R><A><T>3</T><N>OrigStation_NR</N><V>739</V></A><A><T>4</T><N>EventType_NR</N><V>0</V></A><A><T>10</T><N>Id</N><V>1605E3020000470363029A0200000000000000000000</V></A></R>**;AutopilotUNAVAILABLE            ;CAUTION  ;nav/5058                        ;CATEGORY B;0;0;6
26.12.23. 13:02:06 ;26.12.23. 13:00:30 ;OFF  ;LogOnly ;1171 HDG failure                     ;secondary sensor failed Gyro1   ;secondary sensor failed Gyro1   ;5029406703617_G19_O178    ;CAUTION  ;                                ;CATEGORY B;3016;852;7

Extract from my promtail configuration file:

scrape_configs:
  - job_name: alarms
    static_configs:
      - targets:
          - localhost
        labels:
          job: Alarm_logs
          __path__: 'C:\\Logging\\Alarm1.log'
    pipeline_stages:
      - regex:
          expression: '(?s);(?P<ExtendedInfo><[^;]+>);'
      - labels:
          ExtendedInfo: 
      - replace:
          expression: '(<([^;]*?)>)'
          source: ExtendedInfo
          replace: '-'

Here is how the replace stage looks like when promtail is run with the --inspect argument:

[inspect: regex stage]:
{stages.Entry}.Extracted["ExtendedInfo"]:
        +: <R><A><T>3</T><N>OrigStation_NR</N><V>739</V></A><A><T>4</T><N>EventType_NR</N><V>0</V></A><A><T>10</T><N>Id</N><V>1605E3020000470363029A0200000000000000000000</V></A></R>
[inspect: replace stage]:
{stages.Entry}.Extracted["ExtendedInfo"].(string):
        -: <R><A><T>3</T><N>OrigStation_NR</N><V>739</V></A><A><T>4</T><N>EventType_NR</N><V>0</V></A><A><T>10</T><N>Id</N><V>1605E3020000470363029A0200000000000000000000</V></A></R>
        +: ---3--OrigStation_NR--739----4--EventType_NR--0----10--Id--1605E3020000470363029A0200000000000000000000---

And here is what that log entry looks like in Grafana Explore:

xml-like-string
xml-like-string1894×45 41.2 KB

To sum things up:

  • promtail run with --inspect argument shows that replacement takes place
  • the log entry appears in its original form (without replacement) in Grafana.

What am I missing? Any clues will be greatly appreciated!

2

The replace action replaces the source label, unless you leave it empty then it replaces the source log line.

In generally I would recommend you to not tamper with the log line unless absolutely necessary, and instead do the string replacing on Loki.

3

Hi Tony,
Thanks for the feedback. I am not sure how to perform the replacement in Loki, so I fixed the problem by removing the “regex” and “labels” stages and rewriting the “replace” stage as follows:

  - job_name: alarms
    static_configs:
      - targets:
          - localhost
        labels:
          job: Alarm_logs
          __path__: 'C:\\Logging\\Alarm1.log'
    pipeline_stages:
      - replace:
          expression: '(?s);(<[^;]+>);'
          replace: '-- XML dropped --               '  #  or any other replacement string