I have Alerting configured and working nicely. I’d like to have one additional layer to monitor Grafana, to detect if Alerting is down.
Is it enough just to call api/health?
I notice this only does a DB connection check, but maybe that’s enough?
It depends.
Is alerting down if you are not able to send alert email (ms team message, pagerduty alert, …)?
Is alerting down if datasource for alert query is down?
…
Technically, alerting is not down - only some component before/after alerting has a problem. But users may consider that alerting has a problem in this case. So the best option is to test it end to end.
That is a good point, I’ll see if I can configure an always firing alert, pointed at a dead man’s switch
Yes, this known as “Who monitors the monitoring systems?”
Quis custodiet ipsos custodes?