Setup alert when node_exporter is down

[simtcrom]

Hello,

I have several servers with node_exporter running. And in they show up in Prometheus as below,

In my Grafana, Prometheus is already added as a datasource.

Also telegram is added as contact point and tested that its working.

How can I setup a single alert rule in grafana that can send me message to telegram with content

“node_exporter on {{ instance }} is down“ when node_exporter on any of instance is down?

Thanks.

[prashanthan1987]

Hi @simtcrom ,

Below are the steps -

1. Alert rules and create “New alert rules
2. Name the alert rule “Node Down
3. Define query and alert condition
4. Select Metric Name as UP {further filter can be applied and it depend on the use case}
5. Add Expression “Threshold” and Select “Set as alert condition”
6. Select “IS BELOW” and enter value 1 . Do Proview
7. Set evaluation behavior and select the folder where these alert rules should be part of it
8. Create new evaluation group
9. Under Notifications, select Contact Point
10. Under Add annotation
    1. Summary - Node {{ $instance }} is unreachable or down
    2. Description “ you can add as per your need”
11. Save the rule

[simtcrom]

Thank you for responding. I tested this. And alert came as below,

Status: firing
Rule: Node exporter down
Value: 0

I would like get the instance name, somewhere in that message.

Here is a screen shot of the alert I created as you explained. If you could please review and let me know anything I am doing wrong.

[prashanthan1987]

@simtcrom Please copy the same message to description field as summary

[simtcrom]

@prashanthan1987

Changed it and tested and still alert message came as same,

Status: firing
Rule: Node exporter down
Value: 0

[simtcrom]

I also found below logs upon running,

sudo journalctl -u grafana-server.service -e --no-pager

Sep 18 10:32:08 pi5server grafana[6046]: logger=ngalert.state.manager rule_uid=feycqvmxxotfkf org_id=1 t=2025-09-18T10:32:08.340935755+05:30 level=error msg="Error in expanding template" error="failed to expand template '{{- $labels := .Labels -}}{{- $values := .Values -}}{{- $value := .Value -}}Node {{ $instance }} is down': error parsing template __alert_Node exporter down: template: __alert_Node exporter down:1: undefined variable \"$instance\""

[prashanthan1987]

Thanks for sharing screenshots. I am look into it.

[simtcrom]

I was able to finally make it work.

my alert settings as below,

query: ```up{job="node"}```

condition: ``WHEN LAST OF QUERY IS BELOW 1```

Then under ``Configure notification message```

Added two custom annotation as below,

name: prefix

content: {{ $labels.instance }} is down

name: message

content: {{ $labels.instance }} is down

Then created a notification template as below,

{{ define "telegram_node_exporter_down" }}
{{ if gt (len .Alerts.Firing) 0 }}
{{ with (index .Alerts.Firing 0) }}
Status: {{ "\u200B🚨" }} FIRING
Rule: {{ .Labels.alertname }}
Time: {{ .StartsAt }}
Prefix: {{ or .Labels.prefix .Labels.instance }}
Message: {{ or .Labels.message (print .Labels.instance " is down") }}
{{ end }}
{{ end }}

{{ if gt (len .Alerts.Resolved) 0 }}
{{ with (index .Alerts.Resolved 0) }}
Status: ✅ RESOLVED
Rule: {{ .Labels.alertname }}
Time: {{ .EndsAt }}
Prefix: {{ or .Labels.prefix .Labels.instance }}
Message: {{ or .Labels.message (print .Labels.instance " is back UP") }}
{{ end }}
{{ end }}
{{ end }}

Under telegram contact point under optional telegram settings, edit message and select above notification template.

This would now trigger the alert as below,