Datasource Managed Rules never Match Notification Policies

  • What Grafana version and what operating system are you using?
    Grafana Cloud

  • What are you trying to achieve?
    Get notifications for KubePodCrashLooping errors.

  • How are you trying to achieve it?
    Installed Grafana k8s integration helm chart.
    Connected to grafana cloud.
    rules all show up under integrations-kubernetes
    Have a catchall notification policy which is working fine for manually created rules

  • What happened?
    Put stress on the cluster, triggered crash loop backoff. Alert starts firing but notifications are never caught by the default (and only) policy. I have zero tags on this policy and it catches other firing alerts.

  • What did you expect to happen?
    Expected to receive slack/emails for this firing alert.

I’m using the default rules created by the Kubernetes integration.

What is the secret or trick to getting the default catchall policy to send an email or slack when these fire? None of these Mimir/Cortex/Loki rules ever send a notification. Only the “Grafana managed” ones work.

The docs say:

Configure notifications

Add labels to your alert rules to set which notification policy should handle your firing alert instances.

All alert rules and instances, irrespective of their labels, match the default notification policy. If there are no nested policies, or no nested policies match the labels in the alert rule or alert instance, then the default notification policy is the matching policy.

  1. Add labels if you want to change the way your notifications are routed.Add custom labels by selecting existing key-value pairs from the drop down, or add new labels by entering the new key or value.
  2. Click Save rule.

What Preview routing is showing for those rules?

Are used contact points configured correctly? How Show state history looks like?

Never mind, I didn’t realize that there were two separate instances of alertmanager. I configured the second one and these datasource-managed alerts started working. This could be better documented for new users.

The second one on the pulldown menu was the one I had to configure to start receiving alerts.


1 Like