Mute timings for all alerts

Grafana Alerting
1

I can’t add that mute timing to the “Default policy” under “Notification policies”: it’s not available when I edit that policy (no dropdown for timings).

I can click “New nested policy” and add a nested policy without a matcher. This should/might match ALL instances, right?

Is that the correct way to do that? For me it seems not to work, last night there still were alerts sent.

goal: per default alerts should be muted during a defined time window (backups).

used release: grafana-oss-10.1.4 in a docker-compose-stack

2

What are you showing for your mute timings?

3

image
image861×358 31.8 KB

See screenshot. Does that answer your question (I am not 100% sure)?

4

Not sure if this is the issue, but can you set using UTC?

5

Hm, I can try.
I’d appreciate some screenshots of a working set of notification policy and mute timing.

6

changed to UTC. Have to wait until tomorrow to check if alerts were muted.

7

Here’s what I use for one of my integrations:

image
image936×267 24 KB

8

Thanks. Can you also show how the notification policy looks like? And if there is a nested policy etc?

9

This is my policy that has the east coast mute timing. It is under the default policy:

image
image741×707 32 KB

I have a Severity label as well to define different mute timings. Warnings get muted during business hours and critical is 24/7. The severity level is calculated dynamically

10

I don’t have that dropdown “Mute timings”:

image
image748×686 39.5 KB

That policy is within Alertmanager “Grafana”, is that OK?

11

You have to create a new policy. The default policy doesn’t have the ability for mute timing

12

I only see the possibility to create a “nested policy”, right? I had that already and it didn’t work. I will retry asap. Thanks …

13

A new nested policy will do it

14

OK … I try this:

image
image978×540 45.5 KB

15

So the mute policy is there. Now what about the actual alert? What are your labels?

16

Example of what I have:

image
image1352×369 24.8 KB

In your notification labels you would put

Key: job
Value: node

17

The severity is dynamic. Basically it determines if it’s a critical alert or a warning:

{{ if gt $values.C.Value 90.0 }}Critical{{ else }}Warning{{ end }}
18

Thanks for that. It seems I don’t have that yet. Browsing for these “Custom Labels” now, pls specify where I find them, I can’t find it so far. Is it under “Contact points - Notification templates” ?

19

Custom Labels | Grafana Cloud documentation tells me that’s available in Grafana Cloud only.

I run it on-premise.

20

Ah, I see it in the alert rule itself. There are NO Custom Labels there and I don’t need them, as far as I know right now …

The “Preview routing” looks sane to me … it matches my nested policy with “job = node”.

EDIT: added labels to a rule … and trying to figure it out …