How to preserve state between log entries with loki

Grafana Alloy
1

I am using Alloy to process logs generated by an application I do not own and sending them to Loki.
The problem I have is that in the file there is a single line produces some state that needs to be applied to all subsequent logs until another like changes that state.
I think the easiest way to think of it would be that the app logs a line with a request ID, and all logs that follow are associated with that request ID until a new request ID is logged.

I am not sure exactly how I can have a pipeline preserve some state across log entries on a per file basis.

Does that make any sense?

2

I would recommend you to use multiline and group all logs into one line for each request ID (assuming you can reliably tell the start of such a line and that there aren’t too many lines).

If you can provide an example that would be great.

3

Yea the problem is that there would be too many lines. I don’t have an actual example I can share, but conceptually it is something like:

<timestamp> Starting Transaction #1
<timestamp> something a
<timestamp> something b
<timestamp> something c
<timestamp> something d
<timestamp> Starting Transaction #2
<timestamp> something e
<timestamp> something f

The number of actual log lines to be associated with each transaction is huge. I was thinking was perhaps I could have a global attribute which could get set every time I hit a line with a new transaction number?

4

There is unfortunately no such thing.

How many lines are we talking about here? If it’s under a hundred I think you’ll probably be just fine.

5

it’s likely thousands or more.

6

Then you probably will need some external solution. Either:

  1. Your app should just include request ID for every log line.
  2. Or you need something external. Maybe have a simple script that reads the log file, outputs each request ID to a separate file, and have alloy pick them up there. You can then either change the log to inject the request ID, have the ID be part of file name, or something else.
  3. Maybe other logging agent might support, but I can’t think of any at least from my personal experience.

Ideally #1 would be best. I always find it problematic when applications produce log lines that are supposed to be sequential without any identifier. This will break quite easily, say, if the app is multi-threaded or does two requests at once.

1 Like