Loki Query performance reduced after moving parser to alloy agent

Grafana Loki
, ,
1

Hello All

We have a distributed/scalable Loki instance running . we were collecting IIS logs and EKS logs.

Rather than running a parser on All the collected IIS Logs ,we wrote a parser and moved it to Alloy agent . it was parsing the IIS logs into client IP, query ,HTTP Status etc .

We assumed that having All the fields readily available will make the query performance better .

Unfortunately, It got a whole lot worse …

While querying 60 GB of EKS logs returns in 3 secs, while querying 500 Mb of ISS logs takes more than 30 secs .

Is there any suggestion to improve performance or a way around the problem

2

In general you want fewer labels, and the values you decide to parse for labels you want the values to be bounded (see Label best practices | Grafana Loki documentation).

Client IP and query, in your example, definitely aren’t suited to be made into labels.