Mute timing does not work for notification policy

Grafana Alerting
1

Hi everyone,

I have an alert rule and I am trying to apply mute timing on the weekend for some label values only. This is an example of how it is configured currently in notification policies:

  1. First policy matching labels:
    alertname=~dedicated servers.+
    provider!=Provider1
    provider!=Provider2
    This policy does not have any mute timings.

  2. Second policy matching labels:
    alertname=~dedicated servers.+
    provider=Provider1
    provider=Provider2
    This policy has a mute timing applied for whole weekend.

But alerts on the weekend for Provider1 and Provider2 are sent anyway as if first notification policy ignores labels exclusion. I’ve checked matching labels in ‘Alert rules’ tab and don’t see issues with patterns.

How to find out why?

Initially I thought that this is a bug or the problem with SQLite being used, but even on the latest version of Grafana and PostgreSQL database it’s not working.

I think it would be helpful to see in ‘State History’ of alert rules which notification policy was used for alerting.

Thank you.

1 Like
2

Couldn’t find out what was wrong with mutes. Instead I made a script in PowerShell to recreate silencers with scheduled runs (by task manager or cron job) via Grafana API, which seems like a more flexible solution.

#Requires -Version 7.0

Param(
    [string]$url = "http://ip:3000/api/alertmanager/grafana/api/v2/silences",
    [datetime]$silenceStart = (Get-Date).ToUniversalTime(),
    [datetime]$silenceEnd = $silenceStart.AddMinutes(2895),
    [string]$silenceComment = 'weekly silence'
)

Start-Transcript -Path $PSScriptRoot\Grafana_Silencer.script.log -Append

$matchers = @(
	@{'isEqual'=$True; 'isRegex'=$True; 'name'='server'; 'value'='server1|server2'},
	@{'isEqual'=$True; 'isRegex'=$True; 'name'='provider'; 'value'='provder1|prover2'}
)

$body = @{
  createdBy = 'pwsh_runner'
  comment = $silenceComment
  matchers = $matchers
  startsAt = $silenceStart
  endsAt = $silenceEnd
} | ConvertTo-Json

# AllowUnencryptedAuthentication is needed due to http. In local network it may work a while, but it's not good in general.

Invoke-RestMethod -Method 'Post' `
                  -Body $body `
                  -Authentication Bearer `
                  -Token (Import-Clixml $PSScriptRoot\grafana_token.ps1.credential) `
                  -Uri $url `
                  -AllowUnencryptedAuthentication `
                  -ContentType "application/json" | ConvertTo-Json

Stop-Transcript