Mute timings for all alerts

I can’t add that mute timing to the “Default policy” under “Notification policies”: it’s not available when I edit that policy (no dropdown for timings).

I can click “New nested policy” and add a nested policy without a matcher. This should/might match ALL instances, right?

Is that the correct way to do that? For me it seems not to work, last night there still were alerts sent.

goal: per default alerts should be muted during a defined time window (backups).

used release: grafana-oss-10.1.4 in a docker-compose-stack

What are you showing for your mute timings?

See screenshot. Does that answer your question (I am not 100% sure)?

Not sure if this is the issue, but can you set using UTC?

Hm, I can try.
I’d appreciate some screenshots of a working set of notification policy and mute timing.

changed to UTC. Have to wait until tomorrow to check if alerts were muted.

Here’s what I use for one of my integrations:

Thanks. Can you also show how the notification policy looks like? And if there is a nested policy etc?

This is my policy that has the east coast mute timing. It is under the default policy:

I have a Severity label as well to define different mute timings. Warnings get muted during business hours and critical is 24/7. The severity level is calculated dynamically

I don’t have that dropdown “Mute timings”:

That policy is within Alertmanager “Grafana”, is that OK?

You have to create a new policy. The default policy doesn’t have the ability for mute timing

I only see the possibility to create a “nested policy”, right? I had that already and it didn’t work. I will retry asap. Thanks …

A new nested policy will do it

OK … I try this:

So the mute policy is there. Now what about the actual alert? What are your labels?

Example of what I have:

In your notification labels you would put

Key: job
Value: node

The severity is dynamic. Basically it determines if it’s a critical alert or a warning:

{{ if gt $values.C.Value 90.0 }}Critical{{ else }}Warning{{ end }}

Thanks for that. It seems I don’t have that yet. Browsing for these “Custom Labels” now, pls specify where I find them, I can’t find it so far. Is it under “Contact points - Notification templates” ?

Custom Labels | Grafana Cloud documentation tells me that’s available in Grafana Cloud only.

I run it on-premise.

Ah, I see it in the alert rule itself. There are NO Custom Labels there and I don’t need them, as far as I know right now …

The “Preview routing” looks sane to me … it matches my nested policy with “job = node”.

EDIT: added labels to a rule … and trying to figure it out …