Hi, I’m new to Loki and am wondering if it can address our scenario. We have ~150 servers, each generating ~2 GB log files each day. The log file format is like below:
cmd-start, id: 1, timestamp, description, username cmd-start, id: 2, timestamp, description, username cmd-end, id: 1, timestamp, bytesTransferred cmd-end, id: 2, timestamp, bytesTransferred
We would like our log ingestion pipeline to aggregate these entries with the following criteria.
cmd-endevents should be collapsed into a single event. I.e., only one event per
idis ultimately saved.
- The single resulting event should have a new
durationvalue based on the difference between the
timestampsof the start and end events.
- The process should account for interlaced log entries–e.g. see how command “2” shows in the log before command “1” ends.
- The solution should be scalable so that we can tail data in Grafana more-or-less in real-time for all servers.
Logstash from the ELK stack can do these things, but we have concerns about scale with that technology. Can Loki and the PLG stack help? Thanks!