I am working on a larger project for which a number of Grafana alerts have been set up (we’re using Grafana 9.2.1). They’re firing reliably and inform people via email.
I would like to track the behaviour of the alerts over time in different dashboards. I am imagining simply selecting a datasource that contains all the alerts, but I have no idea how this may be possible.
Any help greatly appreciated!
Nobody has any ideas on this? It seems like it would be a need for many people! I am also being asked by many users if this is something we could provide.
Off the top of my head, this could be done via Node-RED, whereby a webhook is sent (with the alert) info and received by Node-RED, which then parses the info (acc. to alert type, server ID, machine, severity, etc.) and inserts into a database of your choice.
Grafana Cloud has *-alert-state-history datasource, which is Loki instance with alert state history log records in json format. You can use LogQL and build query which will meet your needs.
I guess something similar can be done also, when you logs from your on-prem Grafana.
Alerts are also stored as annotations, so you may explore annotation table in your Grafana DB.