Use fired alerts as input data source

I am working on a larger project for which a number of Grafana alerts have been set up (we’re using Grafana 9.2.1). They’re firing reliably and inform people via email.

I would like to track the behaviour of the alerts over time in different dashboards. I am imagining simply selecting a datasource that contains all the alerts, but I have no idea how this may be possible.

Any help greatly appreciated!

1 Like

Nobody has any ideas on this? It seems like it would be a need for many people! I am also being asked by many users if this is something we could provide.

Off the top of my head, this could be done via Node-RED, whereby a webhook is sent (with the alert) info and received by Node-RED, which then parses the info (acc. to alert type, server ID, machine, severity, etc.) and inserts into a database of your choice.

Grafana Cloud has *-alert-state-history datasource, which is Loki instance with alert state history log records in json format. You can use LogQL and build query which will meet your needs.

I guess something similar can be done also, when you logs from your on-prem Grafana.

Alerts are also stored as annotations, so you may explore annotation table in your Grafana DB.

1 Like