Pre-parse or real-time parse

[lovesharepc]

Hello every
I was use ELK before
in ELK, need pre-parse data to field to do data analyze

bug in loki, it’s look use real-time parse to create field

I want to know, which way is good for loki ? pre-parse or real-time parse

I current use way is fluent-bit parse and json format to loki
and loki apply json parser
but this like parse two time, a little strange
does ther have better way to do data analyze in loki ?

[tonyswumac]

I recommend taking a look at Best practices | Grafana Loki documentation.

In general, you should avoid having too many labels. From fluentd you should do as little parsing as possible to get the labels you want / need. It has the benefit of keeping your log pipeline simpler, as well as avoiding have too much cardinality in Loki labels.

[b0b]

A few thought I have had while transitioning from Elastic Stack to Loki.

Just looking at the log message. The labels were already covered

The need for “pre-parsing” the log message is limited with Loki, as the message is not indexed as fields, but rather as just a plain string. Having a well structured log line will however make parsing at query-time a nicer experience.

So pre-parse your log messages, if you need to, so your log messages are easy to parse at query-time. Hope that makes sense.

On a different note, JSON is not a particularly human friendly log format so I’m expecting many of our teams to possibly switch to logfmt as at least I find it easier to read. But who knows…

[chihhaowang]

thank for replay
I think logfmt is a good idear