How do I graph numbers extracted from a log line

Grafana Time Series Panel
1

I’d like to graph the number of readings that a script processes, as well as a few other metrics. All of the values are output on a single line:

2020-11-12 11:08:52 - dsm - INFO: Processed 33000 readings, 1 files, 3827 stations, 0 errors in 10 seconds

I’m able to extract these fields as ‘derived fields’ in Loki. I can even see these values as columns in a table in Grafana. They’re available as ‘parsed fields’ in the Explore view.

What I can’t figure out how to do is how to graph these extracted numbers.

Setting these up as labels in promtail doesn’t seem like the correct approach. That would give me a separate stream for each readings value, and not only does that have high cardinality, it’s not useful.

Outputting a separate line for each reading in the log file would get me what I want, as I could count the number of lines, but that’s a really wasteful approach as I typically process more than 20,000 readings per invocation.

The number of readings is available as ‘NumReadings’ to the Grafana Table View. How do I select that value (33000) to be graphed in a query? Ideally I’d like the number to be graphed against the time it happened.

2

Hi andyvan92117,

I guess you have already found the solution but let me write here my suggestions to help others.
First we need a label selector, e.g. mylabel. Then we need a regexp filter to create labels for those numbers and a max_over_time to convert the lines to actual numbers.
So something like this should work:

max_over_time({mylabel=“myvalue”}
|= “dsm”
| regexp “Processed: (?P[0-9]*)”
| unwrap readings
[5m])

Regards,
Gabor

3

Unfortunately, no, I never figured it out, and this didn’t work for me.

Here’s what I use to count Successful processing:
count_over_time({job=“dsm”} |= “Processed” |= “, 0 errors” [2m])

Here’s the error message I saw when I put in your suggestion:
parse error at line 1, col 24: syntax error: unexpected IDENTIFIER, expecting STRING

I then tried modifying your suggestion to be the following:
max_over_time({job=“dsm”}
| regexp “Processed: (?P[0-9]*)”
| unwrap readings
[5m])

I even tried removing the newlines but that made no difference.

I then got the same parse error, but now on col 20.