Syslog structured-metadata forwarded to loki but not as label possible?

Grafana Loki
1

Hello everyone!
I am currently stuck with some logQL query I want to achieve.

My setup is like this:
a promtail that receive logs through syslog and forward it to loki.

My logs have a structured-metadata with a section eventID=xxxx

I want to create a logQL query that can retrieve this eventID but I don’t want to labelize this eventID as it is high cardinality.

Is there a way to do that?

To be clear I am able to do {eventID=“xxxx”} when I put

relabel_configs:
  - source_labels: ['__syslog_message_eventID']
    target_label: 'eventID'

in promtail config but I don’t want to do that I prefere to do a request like : {appname=“yyyy”} | eventID=“xxxx”

I have tried several things:
-print_structured_data: yes on promtail
-allow_structured_data: true on loki
check documentation, issues on github but I don’t see this use case anywhere.

Don’t hesitate to tell me if you need more info on my config!
thanks in advance!

2

You can alter the log line with something like template (template | Grafana Loki documentation), but in general I try not to do that to avoid potentially messing up the original log format.

You might want to take a look at structured_metadata | Grafana Loki documentation. I haven’t tried this myself, but could be a solution for you.

3

I tried two configuration:

client:
  url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: syslogs
    syslog:
      listen_address: 0.0.0.0:7777
      idle_timeout: 60s
      use_incoming_timestamp: yes
      label_structured_data: yes
    relabel_configs:
      - source_labels: ['__syslog_message_hostname']
        target_label: 'host'
      - source_labels: ['__syslog_message_app_name']
        target_label: 'appname'
      - source_labels: ['__syslog_message_sd_prometheus_1722_level']
        target_label: 'level'
      - source_labels: ['__syslog_message_sd_prometheus_1722_subtype']
        target_label: 'subtype'
      - source_labels: ['__syslog_message_msg_id']
        target_label: 'msgid'
    pipeline_stages:
    - match:
        selector: '{app=~".+"}'
        stages:
        - template:
            source: eventID3
            template: '{.__syslog_message_sd_prometheus_1722_eventID}'

There I’m not able to find eventID3 with a request like : {appname=“xxxx”} | eventID3=~“.+”

I also tried with a pipeline stage like that:

    pipeline_stages:
    - match:
        selector: '{app=~".+"}'
        stages:
        - regex:
            expression: "eventID=(?P<eventID2>\\S+?)"
        - labels:
            eventID2:
        - structured_metadata:
            eventID2:

There I can find eventID2 if the eventID is defined in the body of the logs but not when it’s a metadata.

I also found in the documentation that after the relabeling step all the labels starting with __ are dropped. What I would like is to not drop them but without labelizing them to avoid creating to much streams/chunks.

4

I am not quite sure why you need to find eventID. Presumably all your logs come with an eventID tag, then you should simply either alter the original logline and add the eventID, or assign a key to it in structured_metadata.

If you can provide an example logline with the tags and what your end result should be that would help too.

5

The reason why I need to retrieve the eventID is because I have some alerts based on this ID (thoses alerts are detected by loki forwarded to an alertmanager). This means that at some point I need to do a logQL request of the kind: {appname=“someApp”} | eventID=“130”.

My promtail receive through syslog which mean we have created our own syslog format in the components. Currently the format sent to promtail looks like this:
156 <131>1 2023-11-14T10:27:12.000Z someDNSName someAppName 4444 TooManyAAForEventUid [prometheus@1722 subtype=“33” eventID=“4280” level=“ERROR”] Err transferring files

The problem is that everything that is between “[” “]” is a structured-metadata and the only way I can retrieve the information in loki through a logQL request is to relabelize with a config like that on promtail:

relabel_configs:
- source_labels: [‘__syslog_message_sd_prometheus_1722_eventID’]
target_label: ‘eventID’

The down side with this method is that forces me to put eventID as a label in loki which I don’t want because it is high cardinality and will end with performances issue.

And the problem is that after the step “relabel_config” of promtail, eventID is thrown and loki never sees it.

I have managed to get the eventID on loki only if I write a log like this:
156 <131>1 2023-11-14T10:27:12.000Z someDNSName someAppName 4444 TooManyAAForEventUid [prometheus@1722 subtype=“33” level=“ERROR”] eventID=“4280”, Err transferring files

This is not a correct syslog format, but I managed to retrieve the eventID with a logQL request like this:
{appname=“someApp”} | logfmt | eventID=“130”

This is the behaviour I would like to have but with the syslog log format correct.

I hope it is more clear.

ps. all the pipeline stage are applied after the relabeling steps which means the eventID have already been thrown by promtail.