Is it possible to trigger a Grafana Notification Policy from a Prometheus alert JUST using Labels

[jasonepp]

Is it possible to trigger a Grafana Notification Policy from a Prometheus alert JUST using Labels.

Scenario…

-Grafana configured with a Notification Policy with a label (team=devops)
-AKS with Prometheus configured with some alerts that have the same label (team=devops)

-The Grafana and AKS/Prom are NOT on the same environments They are on separate AKS/VMs.

I want to connect the Grafana Data source to the AKS Prometheus, and WITHOUT EXTRA CONFIG pickup the alert when fired and trigger the notification policy because the labels match.

I want the Prometheus to be agnostic, and just raise its alerts.
I want the Grafana to agnostic and just listen for any Prometheus alerts and send email.

If a new AKS/Prom (with same alerts and labels) is connected to the Grafana, it just picks up the alert and sends email.
If a new Grafana is deployed (with same policy settings), connecting it to the existing AKS/Prom will be enough to pick up alerts and trigger notification policies.

I have some Grafana alerts working off the Prom data source metrics, triggering the policy, and sending emails. But I want to see if I can get the Prometheus Alerts sending emails within Grafana

I’ve have the Prometheus alerts sending the same labels and some Grafana alerts, but the policies are not being triggered.

Using Grafana 11.1.3

T.I.A.

[codi639]

Hey @jasonepp

I’ve never done that but still give it a try. I think you can do it using Contact Point:

Let me know how it goes…

Regards

[jasonepp]

Thanks for your reply.

That config panel looks like a notification policy.
Which I have working for a Grafana Alert.

But can I make it work for a Prometheus Alert.

The Grafana UI shows the Prometheus alert item, and when it fires.

It should be easy to send email on any alert recv’d with minimal/no config on the Prom side for an alert manager. Especially when only a label is required to trigger a Notification Policy from a Grafana Alert. If Grafana is connected to it, send email when alert triggered.

Or, am I missing something?