Using API to silence all alerts with the new Grafana alerting

Grafana Alerting
,
1

I recently switched to the new alering method within Grafana 8.3.3
I used to silence all my alerts with a api call on set times in the week because of server updates to not have false positives.
Now with the new alerting the API does not seem to work.
If i call to silence all alerts for the next 3 hiours i see no silence rule being added in the silence dashboard.
Is there a new API or different API?

  • What Grafana version and what operating system are you using?
    RHEL8 with Grafana8.3.3
  • What are you trying to achieve?
    Pause all alerts with API call
  • How are you trying to achieve it?
    Calling the API from bash in crontab
  • What happened?
    Nothing
  • What did you expect to happen?
    Pausing of all alerts (like before the switch to the new grafana alerting)
  • Can you copy/paste the configuration(s) that you are having problems with?
    n/a
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    n/a
  • Did you follow any online instructions? If so, what is the URL?
    n/a
2

Here is the new API specification for Grafana 8 Unified Alerting:

https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/pkg/services/ngalert/api/tooling/post.json

3

I’m a little bit of a noob perhaps but i don’t really see a replacement API i can do for my API:

curl -f --location --request POST ‘https://myserver:3000/api/admin/pause-all-alerts’
–header ‘Content-Type: application/json’
–header ‘Accept: application/json’
–header 'Authorization: Basic ’
–data-raw ‘{
“paused”: ‘${1}’
}’

4

Were you able to find a way to silence all alerts globally?

5

My latest call goes like this:

APIResp=$(curl --location --request POST ‘https://myserver:3000/api/alertmanager/grafana/api/v2/silences’
–header ‘Content-Type: application/json’
–header ‘Authorization: Basic something’
–data-raw ‘{
“comment”: “prod_env Silence by API without id”,
“createdBy”: “someuser”,
“endsAt”: "’$endSilence’",
“matchers”: [
{
“isEqual”: true,
“isRegex”: false,
“name”: “env”,
“value”: “prod”
}
],
“startsAt”: “’$startSilence’”
}’)

6

That’s going to silence only env=prod, right? I was trying to silence everything.

7

If you have a common field you can use that.
I don’t know if there is a big pause button like before.

8

@waardd I am looking into this as it seems when we put silences in the Grafan UI they are not persistant if the server gets rebooted - so looking to do it via the API.

Couple of questions:
“startsAt”: “’$startSilence’” & “endsAt”: “’$endSilence’”

how have you set these within your call? is it day/month/year time?

I am gathering that the following are just tags on your alerts?

“name”: “env”,
“value”: “prod”

If you could help with this I would be most grateful
Stu

9

Hi! Silences for the Grafana Alertmanager should not be deleted if Grafana is restarted. It would be great to get more information about your version of Grafana and whether you use an external Alertmanager.

1 Like
10

Hi there, thanks for the reply - the version we currently run is:

  • Grafana v8.1.2 (103f8fa094)
    And use Grafana alertmanager
11

Could anyone create a silence with the API?
I’m using Grafana 10.0.1 and the POST for the path /api/alertmanager/grafana/api/v2/silences returns {"message":"silence not found","traceID":""} when trying to create a silence.
If I post with an existing silence values, the POST returns the ID of the existing silence.

12

@fereggert what payload do you send to that endpoint? I think it contains a non-empty id and that can explain the response.

13

You are right @yuriy.tseretyan , I was giving a dummy ID in the payload (I forgot to remove it). Without this property, the API created the silence. Thank you.

14

Any idea on how to create a silence with another timezone than UTC?
Iam using CEST/CET and my first approach was to change grafana server timezone but i couldnt figure out how to do that.