Does Loki support aggregating interlaced multi-line log entries?

Grafana Loki
1

Hi, I’m new to Loki and am wondering if it can address our scenario. We have ~150 servers, each generating ~2 GB log files each day. The log file format is like below:

cmd-start, id: 1, timestamp, description, username
cmd-start, id: 2, timestamp, description, username
cmd-end, id: 1, timestamp, bytesTransferred
cmd-end, id: 2, timestamp, bytesTransferred

We would like our log ingestion pipeline to aggregate these entries with the following criteria.

  • The cmd-start and cmd-end events should be collapsed into a single event. I.e., only one event per id is ultimately saved.
  • The single resulting event should have a new duration value based on the difference between the timestamps of the start and end events.
  • The process should account for interlaced log entries–e.g. see how command “2” shows in the log before command “1” ends.
  • The solution should be scalable so that we can tail data in Grafana more-or-less in real-time for all servers.

Logstash from the ELK stack can do these things, but we have concerns about scale with that technology. Can Loki and the PLG stack help? Thanks!

2

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.