Send every matching log item to email or Slack (or other alert destination)?

Grafana Loki
1

Every time Loki receives a log item that matches a given LogQL query, I’d like to send that line (the full text of it, ideally also with any/all labels) to… an alert destination. In my case, Slack and/or email.

I understand that neither Grafana Alerting nor Ruler can natively do this, as they all operate based on metrics rather than log items.

Can anyone recommend an existing tool that does this… ideally (though not strictly required) with the ability to leverage either Grafana Alerting or Prometheus Alertmanager to send the actual notification?

Failing that, are there any existing “popular” generic solutions for running a LogQL query on a regular interval (say, every minute or every 5 minutes) and then running arbitrary, user-defined code or commands for each matching log item?

2

You can do it with ruler, but it’s not straight forward or particularly neat, but adding the entire logline into a label, something like (not tested):

{SELECTOR} |= "FILTER_STRING" | label_format logline="{{ __line)) }}"

You can now use logline label in your alert body.

3

Thanks, Tony. I haven’t used Ruler before, but I’ll give that a try and see what I can come up with.