Separate alert-rules instead of instances of one alert rule?

Hi,

I’m currently working on implementing Grafana for monitoring our infrastructure. One of our goals is to have Grafana send an alert when a device goes down.

Our current setup includes an alert rule that queries an SQL database containing the hosts and their statuses. An alert is triggered when the status is 0.

SELECT
  status,
  hostname
FROM
  devices

The issue arises when multiple hosts are down; it generates multiple alert instances for a single alert, like this:

image

When this alert appears in Grafana OnCall, we can only acknowledge the entire alert, not the individual instances.

One potential solution is to create a separate alert for each device. However, this approach would require running one SQL query per device, increasing the load on our system.

Is there any other way to have one query that creates a seperate alert per device?

Try to create a notification policy that matches the alert rule (or assign a contact point in rule edit page ) and then in the group_by settings try to group by all labels by using a reserved sequence of .... This will create an alert for every instance.

That works perfectly! Thanks! It still shows up as instances in the regular alert page, but in the OnCall page they show up as individual alerts!

image

1 Like