We recently upgraded to Grafana 8.5 and enabled the new unified alerting system. I’m wondering if there are any best practices regarding the notification policies.
We upgraded from 8.1 so our legacy alerts were migrated to the new alerting system. As part of the migration, each rule got its own notification policy that matches a specific rule uid. Obviously, this isn’t scalable as this would require users to create a new policy each time they set up a new alert.
I was thinking about using labels. For example, define a “channel” label to define which Slack channel the alert should be delivered to. Or, an “owner” to decouple it a bit so we have some freedom to change Slack channels without having to modify all the alerts.
Curious to hear how others have solved this problem.