How to get all alert instances (including Normal state) from alert rule evaluation?

Grafana
1

Context:

Alert rules execute SQL queries that return multiple rows (entities). Grafana creates an alert instance for each row, evaluates conditions, and sends only Firing instances to Alertmanager → Contact Points.

Problem:

I need ALL instances from query execution results - not just Firing, but also Normal/Pending states. Grafana must store this data somewhere between query execution and sending to Alertmanager, otherwise it wouldn’t know what to send.

What I’ve tried:

  1. /api/ds/query - re-executing SQL queries works, but it’s slow and returns duplicates

Questions:

  1. Where does Grafana store alert query evaluation results before sending to Alertmanager?

  2. Is there an internal database/logs with complete evaluation results?

  3. Maybe Grafana IRM or another integration provides access to this data?

Goal:

Build a dashboard showing: alert rule → all monitored entities → their current values and crit. values.

Grafana version: v12.0.1

1 Like
2

The only way to do that is to make all entities be considered as “firing”. Also, you can explore Classic Condition expression that combines all entities into one but it will require some efforts to unpack them back in notifications