Output the entire log line in the Grafana e-mail notification

Grafana Alerting
,
1

Hello everyone,

I use Grafana Loki to analyse logs.

My main application is the notification function.

Example:
In the log file xyz the following content appears ‘could not authenticate’, then send a notification to the defined cantact point.
(sum(rate({filename=‘/var/log/loki/pfSense.log’} |= could not authenticate [1s])))

How can I output the entire content of the log message that triggered this alert in the notification e-mail?

I would be very happy about your help

2

Hi! :wave:

Having a separate alert per log line isn’t recommended.

You also need to calculate the rate over a much longer time frame otherwise this alert will fire and resolve within 1 second, and you most likely will not be notified.

(sum(rate({filename=‘/var/log/loki/pfSense.log’} |= "could not authenticate" [1s])))

I would recommend replacing 1s which a more reasonable value such as 5m or 15m.

You should set up the alert such that it fires when there are log lines with could not authenticate and then this alert links to a dashboard that shows you all of the individual lines.

1 Like
3

Hi,

Thank you very much for your help.

Is it possible that the line from the log is displayed in the notification email?

Best regards

4

This would require having a separate alert per log line, which I wouldn’t recommend.

1 Like
5

I have a similiar problem i have my logs in the grafana through loki as datasourse, there are only two labels on the log {namespace and pod}. I want to filter all the error logs from the loki datasourse with filter {namespace = “something” and linecontains = ‘error’} time range 1min, now i want to send all the error logs to the slack, can i acheive this forwarding of logs to slack through grafana.

Or another functionality i want to imlement is to send seperate message to slack for each error detected in log, and to send the whole log line, can we do this in grafana to forward whole log line to slack.