Logql how-to json function pipe to pattern function

Grafana Loki
1

We use the logging operator to store our logs. With that operator, where possible, we mutate some entries so that those entries are JSON objects consisting of metadata keyed after the metadata names/types and the log object consisting of the actual log meat.

So a small sample of log data might look like:

{
  "kubernetes": "some stuff",
  "cluster_id": "some name",
  "pod_name": "some pod name",
  "log": some log entry generally of an arbitrary serialized or unserialized format depending,
  "namespace": "some namespace",
  etc...
  ...
  ...
  ...
}

In some cases, log is another JSON formatted object and in other cases log is in a different pattern I may want to label-ize.

My question is specifically for the latter case.

Is there a canonical way to apply a pattern() to a json deserialized object? For instance I’ve tried things like:

{query_name="foo"} | json log | pattern `<some> <_> <pattern> <here> | pattern = "thing"`

and

{query_name="foo"} | json log_log | pattern `<some> <_> <pattern> <here> | pattern = "thing"`

fail. Another thing I’m really trying to accomplish with this pragma is finding a rate by some label e.g:

sum(count_over_time({query_name="foo"} | json log | pattern `<some> <_> <pattern> <here>)) by (pattern)`

expecting to get various plots of the rate by the pattern label.

None of this works and I cannot find any documentation on such. It seems as though once something is a JSON object that applying a different function on it (other than regex() perhaps) is not /easily/ doable? :person_shrugging:

(NOTE: I’m not concerned about serialized "log" entries in these scenarios – I’m strictly asking about applying pattern() to a string attribute contained in the JSON "log" object)

2

Can you provide an example of your logs? Specifically the ones where the log field is also JSON formatted.

As to the example you provided, you might try this:

{query_name="foo"} | JSON | line_format "{{.log}}" | pattern `...`
1 Like
3

That actually works! Thank you.

1 Like