Send hostname and IP on alert message

Grafana Alerting
,
1

Hi,

I’m trying to make a notification template that sends the IP and the Hostname of the affected service that is alerting.
When I build the query (a simple query that shows if a service is running, for example), I can’t see the respecive labels with this information. Here’s is an example:

Screenshot_20230327_175359
Screenshot_20230327_1753591445×1300 89.4 KB

What i’m doing wrong? How can I see that info?

2

Hi @tdeltell
Instead of having the query in A and the classic condition in B, can you instead try:

A: query (same as you have now)
B: reduce
C: math expression

Something like this…

image
image1500×575 22.5 KB

3

Thank you! I will try it and post the results

4

I have configured the alert as you told me and, although I already see the labels on the query result (such as host.ip or host.hostname ) i cannot use on the notification. As discussed in this post, I have configured a new annotation called AlertValues with the following content:

{{ with $values }}
{{ range $k, $v := . }}
   IP: {{$v.Labels.host.ip}}
   Hostname: {{$v.Labels.host.hostname}}
   Alerting value: {{ $v }}
{{ end }}
{{ end }}

How can I use that variables on the notification? If i configure the template as in the post mentioned:

{{ define "myalert" }}
  [{{.Status}}] {{ .Labels.alertname }} 
  {{ .Annotations.AlertValues }}
{{ end }}

{{ define "mymessage" }}
  {{ if gt (len .Alerts.Firing) 0 }}
    {{ len .Alerts.Firing }} firing:
    {{ range .Alerts.Firing }} {{ template "myalert" .}} {{ end }}
  {{ end }}
  {{ if gt (len .Alerts.Resolved) 0 }}
    {{ len .Alerts.Resolved }} resolved:
    {{ range .Alerts.Resolved }} {{ template "myalert" .}} {{ end }}
  {{ end }}
{{ end }}

It shows nothing.
Viewing the logs i have the following errors:

logger=ngalert t=2023-04-11T14:50:01.98+0200 lvl=eror msg="error in expanding template" name=description value="{{$v.Labels.agent.name}} CPU is overloaded!" err="error parsing template __alert_CPU Usage Alert: template: __alert_CPU Usage Alert:1: undefined variable \"$v\""
logger=ngalert t=2023-04-11T14:50:01.98+0200 lvl=eror msg="error in expanding template" name=IP value="{{ index $values.A.Labels “host.ip\" }}" err="error parsing template __alert_CPU Usage Alert: template: __alert_CPU Usage Alert:1: unrecognized character in action: U+201C '“'"

Can you help me to continue?

Thank you

5

I solved this using that templates:

{{ define “slack.title” -}}
{{ template “alert_severity_prefix_emoji” . }}
[{{- .Status | toUpper -}}{{- if eq .Status “firing” }} x {{ .Alerts.Firing | len -}}{{- end }} ] || {{ .CommonLabels.alertname -}}
{{- end -}}

{{- define “slack.text” -}}
{{- range .Alerts -}}
{{ if gt (len .Annotations) 0 }}
Summary: {{ .Annotations.summary }}
{{- if index .Annotations “description” }}
Description: {{ index .Annotations “description” }}{{ end }}
{{- if index .Annotations “Value” }}
Value: {{ .Annotations.Value }}{{ end }}
{{ end }}
{{ end }}
{{ end }}

1 Like
6

Hello! Did you solved this?
I cannot retrieve hostname of my servers, could you please elaborate more?