Lables not being attached to Log Data

Grafana Loki
1

Hi all,

I’m trying to attach some labels to my log data from Promtail for Loki. These logs are coming from my FortiGate Firewall and are outputted like this:

Nov 14 03:34:32 172.20.200.1 date=2023-11-14 time=03:34:32 devname="600F-Primary" devid="redacted" eventtime=1699896872275088216 tz="+1000" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="root" srcip=10.10.3.1 srcport=52200 srcintf="port3" srcintfrole="undefined" dstip=23.202.164.20 dstport=443 dstintf="port2" dstintfrole="wan" srccountry="Reserved" dstcountry="Australia" sessionid=336143154 proto=6 action="server-rst" policyid=36 policytype="policy" poluuid="42405eae-e786-51ec-eb3e-c8e961df790a" policyname="Restrictions" user="redacted" group="Restrictions" authserver="FSSO Server" service="HTTPS" trandisp="snat" transip=110.145.195.106 transport=52200 appid=38900 app="Apple.Store" appcat="General.Interest" apprisk="elevated" applist="Restrictions" appact="detected" duration=5 sentbyte=1255 rcvdbyte=12658 sentpkt=17 rcvdpkt=15 shapingpolicyid=16 shaperperipname="Apple IP Shaper" shaperperipdropbyte=0 vwlid=0 wanin=7619 wanout=387 lanin=840 lanout=840 utmaction="block" countweb=1

I am trying to label logs based upon the value defined within the type=“traffic” or type=“utm” to better filter and reduce log query time.

Here is my Promtail Configuration file:

  - job_name: fortigate
    static_configs:
      - targets:
          - localhost
        labels:
          job: "fortigate"
          __path__: /var/log/fortigate/fortigate.log
    relabel_configs:
      - source_labels: ['__raw_log__']
        regex: '.*type=(traffic|utm).*'
        target_label: 'log_type'
        replacement: '$1'

I have attempted to use a Regex Validator online and tried with and without the ’ at the end, the . at the end and the * at the end, but to no avail.

I should see these logs with labels, although all my logs come in as (no unique labels).

What steps should I take to troubleshoot, I’ve asked ChatGPT and still nothing. The systemctl and journalctl logs aren’t indicating anything and the instance reboots correctly.

Any help is appreciated.

2

You need two steps, first step to do a regex group capture, second step to assigned captured value into a label. Something like this (not tested):

pipeline_stages:
  - regex:
      expression: '^.+\stype=\"(?P<log_type>.[^\"]+)\"'
  - labels:
      log_type:
1 Like
3

Hi Tonyswumac,

You are so right!
I eventually tried out a different method, which mimics the label tag after the definitions/expression.

If anyone else stumbles across this, this is what I did:

- job_name: fortigate
  static_configs:
    - targets:
        - localhost
      labels:
        job: "fortigate"
        __path__: /var/log/fortigate/fortigate.log
  pipeline_stages:
    - logfmt:
        mapping:
          type:
          subtype:
          policyid:
    - labels:
        type:
        subtype:
        policyid:
1 Like