Grafana (Cloud version) doesn't send notifications when Alert rule is fired

Grafana Alerting
1

I’ve started using Grafana Cloud today, and it wasn’t really easy to setup.

I’m using it to monitor a Cloudron instance (Ubuntu 22.04).
I managed to get the instance’s data using the “Linux Server” integration, but I couldn’t get any alert to be sent, couldn’t test the alerting system either, I really don’t understand how it works.

I managed to make a rule “fire”, but that didn’t send anything either.

I find the learning experience very slow/difficult for something so common.

Here is a video where I try to show all the relevant information related to the whole thing:

Grafana looks exactly like what I need, but I’m really struggling with that last step of sending notifications whenever something bad happens.

2

Too long, didn’t watch. But my recommendation:

3

Thanks for the link.

Here is my alert rule:

image
image1920×2331 178 KB

It was created automatically when I configured the “Linux server” Integration.
What I understand now, is that those alerts weren’t created using the new “Simplified Routing”, I don’t have the option of selecting a “Contact Point” when editing the alert.

But, I’m kind of back to the beginning: I don’t understand how the system is configured for my alerts.

image
image2856×1840 408 KB

image
image2808×1766 407 KB

image
image2846×1002 222 KB

{
  "template_files": {},
  "template_file_provenances": {},
  "alertmanager_config": {
    "route": {
      "receiver": "grafana-default-email",
      "group_by": [
        "grafana_folder",
        "alertname"
      ],
      "routes": [
        {
          "receiver": "grafana-default-email",
          "object_matchers": [
            [
              "__grafana_autogenerated__",
              "=",
              "true"
            ]
          ],
          "routes": [
            {
              "receiver": "grafana-default-email",
              "group_by": [
                "grafana_folder",
                "alertname"
              ],
              "object_matchers": [
                [
                  "__grafana_receiver__",
                  "=",
                  "grafana-default-email"
                ]
              ]
            }
          ]
        }
      ],
      "group_interval": "1m"
    },
    "receivers": [
      {
        "name": "grafana-default-email",
        "grafana_managed_receiver_configs": [
          {
            "uid": "adq8hacdtapdsb",
            "name": "grafana-default-email",
            "type": "email",
            "disableResolveMessage": false,
            "settings": {
              "addresses": "admin-it@unly.org",
              "singleEmail": false
            },
            "secureFields": {}
          },
          {
            "uid": "adq9fs7tnljb4d",
            "name": "grafana-default-email",
            "type": "webhook",
            "disableResolveMessage": false,
            "settings": {
              "httpMethod": "POST",
              "url": "https://n8n.unly.org/webhook-test/072f54ee-272d-47c7-a32d-df86c01e8e37"
            },
            "secureFields": {}
          }
        ]
      }
    ]
  }
}

image
image1726×1840 307 KB

4

Your alert should have preview routing, where you can see which notification policy is catching your alert.
Unified (not old obsolete legacy) alert is event based - firing event generates event with labels and then based on labels you define which contact point will be triggered in notification policy - that’s complex, but powerful concept (but many people has a problem to understand it, so that simplified routing was also introduced), e.g. os=linux => send email to linux support group, priority=p1 => send text message to CTO, …

5

My issue is that I can’t make the connection between the alert rule and the notification, I don’t understand where to go to analyse/debug the behavior.

I made a shorter video showing only the alert’s configuration:

The alerts have been created using a special configuration, and are “Data source managed”:

image
image535×604 38.8 KB

But I still don’t understand how things are related/configured.
Typically, I don’t understand where to say if os=linux then send email to X.

6

I think I’ll just drop Grafana.

I managed to do what I wanted with Netdata, within 30mn from discovery to testing my system.
It was just so much simpler, I didn’t have to spend hours trying figure out how to send the data from my server, just a single command line and that was it. (unlike configuring Allow which was so not straightforward to understand)

And the notifications worked just straight out of the box, too.

So, for my use case (monitoring 3 Cloudron servers), Netdata seems like a much better fit, simplicity wins.

1 Like
7

Hi! :wave:

It looks to me, from the screenshots and videos, that you might have confused Datasource Managed Alerts and Grafana Managed Alerts.

Your alert rules are Datasource Managed Alerts, as can be seen from the screenshots.

However, you have configured your contact points and notification policies for the Grafana Alertmanager, which is used with Grafana Managed Alerts. In other words – the wrong Alertmanager I’m afraid.

You need to change Alertmanagers from the “Choose Alertmanager” dropdown and choose the Grafana Cloud Alertmanager from the dropdown, the set up your contact points and notification policies there.

I hope this helps!

2 Likes