Multiline feature configuration question

Grafana Loki
1

We just upgraded to latest versions of both loki and promtail (v.2.2.0), mainly to have the multiline feature on stacktrace (exceptions) appearing nicely in loki/grafana and in one log block. However by testing the feature, we noticed two issues:

config example:

snippets:
    pipelineStages:
      - cri: {}
      - multiline:
          firstline: '^\d{4}-\d{2}-\d{2} \d{1,2}:\d{2}:\d{2}.\d{3}'

the regex matches the start of the stacktrace correctly:

2021-03-25 09:26:07.499 [-][-][-][ INFO] [,]...
  • while the first block with the timestamp appears as one log line in Loki/Grafana, every line of stack trace is next to each other instead of a new line, thus making really hard to read it. Is there way to make every line of stack trace to appear in a better format ie more readable?
  • the “caused by” block of the same exception stacktrace appear as separate log line in Loki/grafana, even though is shouldn’t since it belongs to the previous block.

FYI, everything else is kept as default for the multiline: e.g. max_lines: 128 and max_wait_time: 3s.
regarding the second issue, 128 lines haven’t been exceeded to cause this.

Thank you for your work!

2

Solved eventually, when I defined in promtail: - docker:{} in the pipelineStages, instead of cri:{}, exceptions handling started working fine again and appear fine in Loki/Grafana, also new lines in the stacktrace but in one log message as expected.

3

Hi @ckailantzis
I’m just trying to setup similar thing, but I’m new to Promtail and Loki…
Having issue with formatting pipiline_stages for a job.

Could I ask you to share a small example of how you use these snippets?

4

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.