Hello,
using Grafana alerts with Threema Gateway to monitor a self scripted node exporter for some services throws the following alert every four hours:
[FIRING:1] DatasourceNoData Alert rules (fdi74e1dtwkcge A Service failed)
Message:
**Firing**
Value: [no value]
Labels:
- alertname = DatasourceNoData
- datasource_uid = xxxxxxxxxxxxxx
- grafana_folder = Alert rules
- ref_id = A
- rulename = Service failed
Annotations:
- summary = Service failed
Source: http://localhost:9091/alerting/grafana/xxxxxxxxxxxxxx/view?orgId=1
Silence: [http://localhost:9091/alerting/silence/new?alertmanager=grafana&matcher=alertname=DatasourceNoData&matcher=datasource_uid=xxxxxxxxxxxxxx&matcher=grafana_folder=Alert+rules&matcher=ref_id=A&matcher=rulename=Service+failed&orgId=1](http://localhost:9091/alerting/silence/new?alertmanager=grafana&matcher=alertname%3DDatasourceNoData&matcher=datasource_uid%3Dxxxxxxxxxxxxxx&matcher=grafana_folder%3DAlert+rules&matcher=ref_id%3DA&matcher=rulename%3DService+failed&orgId=1)
URL: http:/localhost:9091/alerting/list
This seems to me like a bug because the silence URL is not correct here (other messages have the correct URL https://my-fqdn.tld/…) and silencing does not show any effect. The alarm is fired again though.
Prometheus shows that the used node exporter works fine. The configuration of the alert is attached (did not work because i’m a newbie, so i had to post it inline).
It would be great if you could help me stopping these alerts.
Best regards
Andreas
apiVersion: 1
groups:
- orgId: 1
name: daily
folder: Alert rules
interval: 1d
rules:
- uid: fdqf1l3qi4idcd
title: Reboot required
condition: C
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: xxxxxxxxxxxxxx
model:
editorMode: code
expr: node_reboot_required
instant: true
intervalMs: 1000
legendFormat: __auto
maxDataPoints: 43200
range: false
refId: A
- refId: C
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
conditions:
- evaluator:
params:
- 0
- 0
type: outside_range
operator:
type: and
query:
params:
- C
reducer:
params: []
type: last
type: query
unloadEvaluator:
params:
- 0
- 0
type: within_range
datasource:
type: __expr__
uid: __expr__
expression: A
intervalMs: 1000
maxDataPoints: 43200
refId: C
type: threshold
noDataState: OK
execErrState: Error
for: 0s
annotations:
summary: Reboot of VM is required after automatically APT updates
isPaused: false
notification_settings:
receiver: grafana-default
- uid: xxxxxxxxxxxxxx
title: Service failed
condition: C
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: fdi74e1dtwkcge
model:
disableTextWrap: false
editorMode: builder
expr: node_services
fullMetaSearch: false
includeNullMetadata: true
instant: true
intervalMs: 1000
legendFormat: __auto
maxDataPoints: 43200
range: false
refId: A
useBackend: false
- refId: C
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
conditions:
- evaluator:
params:
- 0
- 0
type: outside_range
operator:
type: and
query:
params:
- C
reducer:
params: []
type: last
type: query
unloadEvaluator:
params:
- 0
- 0
type: within_range
datasource:
type: __expr__
uid: __expr__
expression: A
intervalMs: 1000
maxDataPoints: 43200
refId: C
type: threshold
noDataState: NoData
execErrState: Error
for: 0s
annotations:
description: ""
runbook_url: ""
summary: Service failed
labels:
"": ""
isPaused: false
notification_settings:
receiver: grafana-default