Loki 0.4.0 querying Metrics From Logs

Grafana Loki
1

Dear Grafana Community!
I am quite new in the world of Grafana and I am looking for showing metrics from log content parsed with Promtail to labels.
Did I get it right from the newsletter of the Loki 0.4.0 that via new Prometheus compatible API it is possible to do it?


If it is not necessary I would like to keep the tools I am using as minimum it is possible: Promtail, Loki, Grafana.

My use case would be to get out/parse metrics from log lines and show on Grafana. E.g. in every 10 seconds a new log line is appearing in log file showing some characteristics of the system like free memory, network usage, etc.
I am already able to parse the log lines with Promtail as labels and send to Loki.
Is is possible to reference only the labels by its name to show the values over time?

Thanks,
Tamás

2

It is possible with Grafana 6.4 - you add Loki as a prometheus data source. See this video for a demo - around 1:30.

Support in the UI in Grafana for metrics from Loki logs just got merged: https://github.com/grafana/grafana/pull/20158 and will be included in the next release (6.6) in January.

There are packages and docker containers for the latest master build if you want to test it out now: https://grafana.com/grafana/download/6.6.0-687351e1pre

3

Hi Daniellee,
Thanks for the response! I have also checked the video but as far as I have understood the example it is possible to count the number of logs/second matching on a query.
My use case would be to show the value of the label I am choosing e.g.:
2019-11-18 17:39:01.000 memory_free 15
2019-11-18 17:38:01.000 memory_free 14

If I add to the Query section e.g. {memory_free=“15”} it selects the right entry from the Loki but the return object will contain 2 entries:

  • stream object with all of the labels using : “” form
  • values array of timestamp and whole log line.

How can I add to the graph the labels from stream object? :slight_smile:

Thanks and best regards,
Tamás

4

Assuming that you created a prometheus data source that points to your Loki instance - have you tested with the rate function. Does something like this work:

sum(rate({your_label="a_value"}[1m])) by (memory_free)

5

Thanks Danielle!
It seems I have misunderstood something regarding Loki.
I wanted to extract metrics from the logs and show it in Grafana while I am leaving out Prometheus from the game. As I have learnt the two tools shall work together efficiently. :slight_smile:

Thanks a lot for your help again!

Best regards,
Tamás

6

I have the exact same question but now on Loki 2.0.0. Is it possible to graph e.g. average cpu with Loki datasource and log lines such as:

2020-12-03 15:51:01 cpu_usage=3.89

7

Yes!

avg(avg_over_time{app="foo"} | regexp `.*cpu_usage=(?P<cpu_usage>.*)` | unwrap cpu_usage[1m])

or to average by a label:

avg by (pick some labels) (avg_over_time{app="foo"} | regexp `.*cpu_usage=(?P<cpu_usage>.*)` | unwrap cpu_usage[1m])
  • These queries may not be 100% correct but hopefully gives you the idea, you may need to adjust your regex to match better.

Take a look at the logql docs or the release blog post for some more info!

1 Like
8

Awesome @ewelch, thank you!

9

How to plot a graph in grafana using a value which is present in my log.
My log looks something like this.
INFO 2021-05-15 09:15:30,752 - Greet-Client is Started , duration=0.003822803497314453s
I want to keep duration as my Y-axis. Can someone show some light on how to do this.
I have loki as my data source and these logs are coming from my Kubernetes app
Thanks in advance!!
@ewelch