How to error condition depending on week old logs except logs between time hours?

Grafana Loki
1

I am monitoring some lines in grafana loki and our on-premise grafana installation and I want to write an alert.

An alert should be sent, if: a log line matching a regex in the last hour has happened more time than the average time it has happened for the past week between hours 10AM to 17PM.

I find it impossible to “filter time hours” in grafana alerts gui. I can query week old logs, but how do I restrict log calculation to only log lines between some hours during the day?

Thank you.

2

I do not know if this is possible in Loki. I don’t even know if this is possible with PromQL (which is a lot more flexible than LogQL) if you use a recording rule and send metrics to Prometheus.

For this kind of thing it might be better off to write your own script using API calls.

1 Like
3

I had some fun with the following:

| format_label timestamp_HH=`{{ __timestamp__ | date "15" }}`
| timestamp_HH > 10 | timestamp_HH < 17

But it was not possible to use toDateInZone on __timestamp__ to get the date in Local timezone. The

{{ __timestamp__ | date "2006-01-02T15:04:05.999999999Z" | toDateInZone "2006-01-02T15:04:05.999999999Z" "EST" | date "15" }}

Still assigns the label to the hour in UTC.

Bottom line, I started writing a Python program to iterate over days and collect the data.

Thanks

4

It looks like this issue: [Question] log queries | line_format a timestamp into Local date string · Issue #11732 · grafana/loki · GitHub

1 Like