Display error logs in alert notification for Loki

Grafana Configuration
,
1

  • What Grafana version and what operating system are you using?
    grafana/grafana:9.5.3
    grafana/loki:2.6.1

  • What are you trying to achieve?
    Set up Slack alert notifications in Grafana that include the error logs in them.

  • How are you trying to achieve it?

Following this documentation: Alerting and Recording Rules | Grafana Loki documentation
I have an alert rule with the following query, using a Loki data source:
rate({app=“agones”} |= error [$__interval])

image
image1182×571 52.3 KB

Then, I create a contact point with a webhook URL to a Slack channel:

Title:

{{ len .Alerts.Firing }} firing, {{ len .Alerts.Resolved }} resolved

In the text body, that is where I would like to display the error log.

  • What happened?

The alert notification displays in Slack, but I can’t figure out how to display the error message from the logs.

  • What did you expect to happen?

I would like to figure out a way to get the logs to display in the alert notification.
I have followed this similar topic: Including full log error message in alert notification using Loki

A work around has been suggested to generate a label based on a pattern query

pattern `<_> <_> <log_message>`

Even though I have tested several ways to get this query to work, like replacing the ` with ", but I am always getting an error message in Grafana:

Failed to evaluate queries and expressions: failed to execute query A: parse error at line 1, col 79: syntax error: unexpected STRING

I would appreciate any insight on this.
Thank you.

2

Show us the contents of the “A” query that is giving you the error.

1 Like
3 
sum by(log_message) (rate({app="agones"} |= `error` | pattern `<_> <_> <log_message>` [$__interval]))

Actually, I noticed that it work with Grafana 10+
Earlier versions of Grafana return the String error mentioned in my thread.

1 Like