Does loki/promtail allow rewrite label using internal cache

Grafana Loki
,
1

For example, we have log lines here:

{"timestamp":"Apr 10 03:34:00", "id":"01e8", "message":"---522430890385b4e81f1000d6289ef675---"}
{"timestamp":"Apr 10 03:34:01", "id":"01e8", "message":"jajaja"}
{"timestamp":"Apr 10 03:34:02", "id":"01e8", "message":"bye-bye"}

Promtail pipeline labels result for first message is:

timestamp: "Apr 10 03:34:00"
id: "01e8"
id2: "522430890385b4e81f1000d6289ef675"

For second and third:

timestamp: "Apr 10 03:34:01"
id: "01e8"

timestamp: "Apr 10 03:34:02"
id: "01e8"

Does it’s possible get this behavior:
If “message” match id2 regexp add for current stream id2 label (what actually is done) and for each subsequent stream add id2 label according value of id label?

I know, i can reach that using two queries, first to retrieve id using known data, and second with stream selector using id label. But, some software supports only one “harcoded” query, which doesn’t get all log lines. For example, data which known for some software is “522430890385b4e81f1000d6289ef675” line:
logQL
{job=some-app} |~ "522430890385b4e81f1000d6289ef675"

but purpose is get all log lines with id eq “01e8” (with one query =))).

2

up :=)

3

I don’t think it’s possible with promtail (or any log agent for that matter, without at least some sort of custom script).

However, sounds like you are confident that the logs between two IDs denoted by ---<ID>--- will always belong together, perhaps try to use multiline configuration and group all logs from the same ID into one? provided you don’t have too many log lines inbetween, of course.

For example, given raw logs:

{"timestamp":"Apr 10 03:34:00", "id":"01e8", "message":"---522430890385b4e81f1000d6289ef675---"}
{"timestamp":"Apr 10 03:34:01", "id":"01e8", "message":"jajaja"}
{"timestamp":"Apr 10 03:34:02", "id":"01e8", "message":"bye-bye"}

With multiline configuration (not tested):

- multiline:
    firstline: '^.*"---.*---".*}$'
    max_wait_time: 1s
    max_lines: 10
- json:
    expressions:
      timestamp:
      id:
      message:
- output:
    source: message

Produces results:

timestamp: "Apr 10 03:34:00"
id: "01e8"
output: "---522430890385b4e81f1000d6289ef675---
jajaja
bye-bye"
4

Thanks, for reply ))

5

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.