How to add meta data to log line

Grafana Loki
#1

Hey ppl, can I get your opinion on the best approach here? We have a multi-tenant service for customer devices and working on using Loki to store their device logs. The device logs could be json, logfmt or just text and we would like our tenants to be able to use most features of logQL. However we also would like to associate some extra meta data with each message, such as version , isStdErr , isSystem and serviceId . The options I am considering are:

  1. Store meta data as labels but this would push cardinality limits.
  2. Try parse the log line to add our metadata before writing to Loki. However, this seems error prone and resource intensive. First try parse as JSON then try as logfmt and if plain text reformat the message to logfmt to add the meta data.
  3. Store each log line as JSON with metadata properties and a message property containing the device log. However I think it would require all queries to line_format the message property to search with in the message. e.g. {device_id="1"} | json | line_format "{{.message}}" | ...user query
  4. … something else I haven’t thought of?
#2

I think I would avoid option 1 for the reason you mentioned.

Option 3 I think would be my pick, but yeah this does create a burden for the user, having a dashboard could help but for adhoc queries you’ll be filtering out this metadata a lot.

I would see if you could append the message first before the metadata, and consider using logfmt instead of json, at least then it could be

msg="some existing log message in wahtever format" version="1.0" serviceId="2342432dafefe"

And then you might not even care that much about having to filter away the metadata for adhoc queries

However as I’m typing this I don’t think Promtail can properly add to either json or logfmt right now, is this how you were looking to modify the line?

#3

Option 3 I think would be my pick, but yeah this does create a burden for the user, having a dashboard could help but for adhoc queries you’ll be filtering out this metadata a lot.

To start with log search will be done through our own UI so we can abstract away formatting in both the query and the results. Down the line we would probably look at allowing customers to use logQL directly.

We are pushing logs from our backend into Loki so don’t need Promtail support.

I think this is a good use case for Loki to support adding metadata/pre-parsed fields to EntryAdapter.