Synthetic checks do not trigger oncall alerts

Grafana Cloud
1

My usecase is simple:

  • I have a synthetic check on an endpoint
  • When my synthetic check fails, I want to get a notification through Grafana Oncall

So far what I have is:

  • I setup the Grafana Oncall, when I send a test alert there I get a Slack notif and a phone call, just like I wanted, so it works.
  • I setup the contact point to be Grafana Oncall under the Alerting tab.
  • I have created the synthetic check, it works.
  • I created default alert rules.
  • When a check fails, the alert rule is marked as “Firing”.

However, I do not get any notifications when this rule is firing. What am I doing wrong?

2

I figured it out by myself, in case anyone else runs into the problem:

  • The fact that there’s two alert managers is very confusing.
  • The synthetic check alerts are created in the grafanacloud-<team>-ngalertmanager one, which means you need to configure the notification policies for this specific alert manager instance.
  • When you try to configure the contact points there’s no integration with Grafana Oncall for alert manager, even though it exists for the other alert manager named Grafana, anyway.
  • Then you need to go to your Grafana Oncall, add an integration for “Alertmanager”, which then will give you a URL, copy that.
  • Now go back to the contact points for the grafanacloud-<team>-ngalertmanager one, add a new integration for “Webhook”, then paste this new URL.
  • Then go to “Notification Policies”, and change the default notification policy to contact this new contact point you just created.

Bam, now your alerts are gonna trigger Oncall processes. From then on, you can configure who gets alerted when, by which methods, etc. Overall a very confusing process.

1 Like
3

@burakkarakan1 I apologize this use case was unintuitive for you and your team. I totally agree that the two alert managers are at the root of the confusion for this issue. We are currently working to eliminate this challenge, but the work is slower than we want.

I’m glad you were able to figure it out. We will keep working to make it more intuitive.

4

@burakkarakan1 thanks for this - I just wasted 3 hours debugging this crazy situation myself as well and found this post while searching to confirm my understanding…

Next challenge is how to get the contact point and notification policies on the “hidden” ngalertmanager created via Terraform? I don’t see any obvious option in the provider docs for how to get terraform to choose which AM instance it is configuring.

I don’t suppose you’ve found any solution to that or @marcchipouras has any insight?