Alert if contact point is failing

Hello community,

I set up a contact point of type webhook, redirecting to a custom http server where routing is done with the labels of the alert. It works great, but if labels are incorrectly provisioned, the delivery fails. In the contact point page, we can see that the last delivery fails, but are we able to have some kind of email alert in this case? We don’t want to have alerts that are not delivered to our alerting system because of a provisioning failure.

Thank you for your help,

The only thing that comes to my mind is to write Grafana logs to a store and then query it. When notification fails there is log message Notify for alerts failed emitted by logger ngalert.notifier.alertmanager

Hello Yuriy,

So except using an external alert manager to check this failure, there is no possibility to do this inside Grafana? I was hoping to configure mail as a backup if the custom webhook fails for whatever reason.

I can see two options:

  1. One that I mentioned above. Ship Grafana logs to Loki (or other place that is supported by Grafana). Then Create an alert rule that queries the data source for logs Notify for alerts failed and send it to an email.
  2. Use infinity data source, setup to query Grafana API. Here is an example rule that I managed to run in my environment.
{
    "apiVersion": 1,
    "groups": [
        {
            "orgId": 1,
            "name": "Default",
            "folder": "Grafana Alerting",
            "interval": "1m",
            "rules": [
                {
                    "uid": "eeazviqm7uwowd",
                    "title": "Failing Integrations",
                    "condition": "A",
                    "data": [
                        {
                            "refId": "A",
                            "relativeTimeRange": {
                                "from": 600,
                                "to": 0
                            },
                            "datasourceUid": "ceazv0jwh9ts0b",
                            "model": {
                                "columns": [],
                                "computed_columns": [
                                    {
                                        "selector": "1",
                                        "text": "value",
                                        "type": "string"
                                    }
                                ],
                                "datasource": {
                                    "type": "yesoreyeram-infinity-datasource",
                                    "uid": "ceazv0jwh9ts0b"
                                },
                                "filterExpression": "lastNotifyAttemptError != \"\"",
                                "filters": [],
                                "format": "table",
                                "global_query_id": "",
                                "instant": true,
                                "intervalMs": 1000,
                                "maxDataPoints": 43200,
                                "parser": "backend",
                                "refId": "A",
                                "root_selector": "(\n  $map($, function($item) {\n    $merge([\n      $map($item.integrations, function($integration) {\n        $merge([\n          {\n            \"name\": $item.name,\n            \"integrationName\": $integration.name,\n            \"lastNotifyAttemptError\": $integration.lastNotifyAttemptError\n          }\n        ])\n      })\n    ])\n  })\n)",
                                "source": "url",
                                "type": "json",
                                "url": "http://localhost:3000/api/alertmanager/grafana/config/api/v1/receivers",
                                "url_options": {
                                    "data": "",
                                    "method": "GET"
                                }
                            }
                        }
                    ],
                    "noDataState": "NoData",
                    "execErrState": "Error",
                    "annotations": {},
                    "labels": {},
                    "isPaused": false,
                    "notification_settings": {
                        "receiver": "Black hole"
                    }
                }
            ]
        }
    ]
}

Make sure you change the URL to point to public Grafana URL