Logql optimization

Grafana Loki
1

Hi there,

I have loki logs something like below,

test log pushed - executionTime: 12
test log pushed - executionTime: 20
test log pushed - executionTime: <some_random_integer>

I’m creating a guage panel in Grafana which has average execution time,

I have query like,

avg_over_time({application=“papi”}
|= “executionTime”
| regexp “executionTime: (?P<execution_time>[0-9]+)”
| unwrap execution_time [$__auto]
)

I came across - using regex results results in high cpu consumption and impacts loki performance, can anyone suggest the best possible way we can make it more optimistic query,

Many thanks,

2

You could try pattern and see if it’s better for your memory usage: Log queries | Grafana Loki documentation

3

@tonyswumac , thanks for responding, I tried it pattern parser, however I’m seeing below difference,

when we query in explore tab -

image
image3099×1258 287 KB

when we run query in dashboard panel -

image
image1838×1165 194 KB

I see execution time value is getting appended with new line character, this results in unwrap throwing error while converting it to float, we can get rid of this by adding pattern
| pattern “<_> executionTime: \n”

why are we seeing this difference,and not in explore logs? is it known behaviour? how can we eliminate this kind of characters, in our pattern expression,

Many thanks,

4

That would be from your logs. Loki doesn’t inject any newline character, nor does Explorer.

If you don’t know when your logs would have the newline character, then probably switching back to regex would be better.

In general, Loki’s performance comes from distribution. If you are not running single container, you should make sure query splitting is working. If you are using single container, perhaps consider migrating to simple scalable mode so you can scale if needed.

5

thanks for responding,
assuming we have tons of logs matching executionTime and scraping them as a label to perform metric query, does this effect cardinality or indexing anyway?

I’m just trying to understand this way like,which has advantage -

  1. pushing executionTime label directly to loki → we can use this label to perfrom metric query
    or
  2. not adding this as a label and perform log level operation to scrape this label and performing operation on it.

please suggest your insights,

thanks again

6

Adding to last question,
can we use executionTime as a structured metadata while pushing logs to loki by avoiding indexing?

7

Yes it does. It is recommended to have a “bounded” values for labels, and if you use execution time which can be any floating number it will impact your query performance.

I would recommend real time parsing for things like this.

8

You could, yeah. Structured metadata might be a good usecase for this.

9

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.