Alert notifications - Parsing through the output of the alert to display "only" the error log message

Grafana Alerting
,
1

Hello,

I am trying to send a notification to Slack, when my alert triggers (alert query details at the end).
My alert triggers when it catches an error in the logs (from a loki data source).

In the contact points, I am trying to understand what I can collect and display from the alert itself, by putting this:

{{ range .Alerts.Firing }}

Here is everything: {{ . }}

{{ end }}

This displays the following output when I get 1 alert:

Here is everything: {firing alertname=Agones errors, grafana_folder=log-monitor description=Trying to display the log_message: {{ $log_message }}., summary=Here are the labels related to the alert: __alert_rule_namespace_uid__=DYYJLTgSk, __alert_rule_uid__=E2vxYTRSz, alertname=Agones errors, grafana_folder=log-monitor. 2023-08-22 21:51:30 +0000 UTC 0001-01-01 00:00:00 +0000 UTC http://localhost:3000/alerting/grafana/E2vxYTRSz/view?orgId=1 1661f2d935f5cc1b http://localhost:3000/alerting/silence/new?alertmanager=grafana&matcher=alertname%3DAgones+errors&matcher=grafana_folder%3Dlog-monitor&orgId=1   map[B0:1] [ var='B0' metric='Value' labels={log_message=http: TLS handshake error from 172.29.0.0:4307: EOF} value=1 ]  }

I would like to understand how to navigate through this output and display something that may be a little easier to read, like:

At 2023-08-22 21:51:30 +0000 UTC 0001-01-01 00:00:00 +0000 UTC

error log: http: TLS handshake error from 172.29.0.0:4307: EOF

View it in Grafana: http://localhost:3000/alerting/grafana/E2vxYTRSz/view?orgId=1 1661f2d935f5cc1b http://localhost:3000/alerting/silence/new?alertmanager=grafana&matcher=alertname%3DAgones+errors&matcher=grafana_folder%3Dlog-monitor&orgId=1

The query to generate the alert is as follow:

sum by(log_message) (rate({app="agones"} |= `err` | pattern `<_> <_> <log_message>` [$__interval]))

Then, I generate an alert whenever this sum is > 0:

image
image1312Ă—683 43.3 KB

That is how I am generating a label: log_message, so that I am hopeful to use that label in the alert notification.

Thanks for helping me out with this.

2

Hi! The reference might be of help? Reference | Grafana documentation

1 Like
3

It did help! I was able to reduce the amount of details to something smaller with ValueString. It is not perfect, but it is much better than having to display everything.

Thank you.

4

@georgerobinson i read the reference you provided but didn’t quite understand i am facing same kind of problem in which i only want to include a specific part of the message lets say message in double quotes “” how can i really do that. any example would be really helpful.

here is the template i am using

{{ define “zulip.print_alert” -}}

{{ range .Labels.SortedPairs -}}

{{ if and (not (eq .Name “grafana_folder”)) (not (eq .Name “group”)) (not (eq .Name “job”)) (not (eq .Name “logstream”)) }}

  • {{ .Name }}: {{ .Value }}

{{ end }}

{{ end -}}

{{ if .DashboardURL -}}

Go to dashboard: {{ .DashboardURL }}

{{- end }}

{{- end }}

{{ define “zulip.message” -}}

{{ if .Alerts.Firing -}}

{{ len .Alerts.Firing }} firing alert(s):

{{ range .Alerts.Firing }}

{{ template “zulip.print_alert” . }}

{{ end }}

{{ end }}

{{ if .Alerts.Resolved -}}

{{ len .Alerts.Resolved }} resolved alert(s):

{{ range .Alerts.Resolved }}

{{ template “zulip.print_alert” .}}

{{ end }}

{{ end }}

{{- end }}