Pause all alerts - for a group of hosts

Grafana Configuration
1

Hello,

As there seems to be no way of silencing alerts within the UI, I have just started to look at the possibility of setting up a “Silence” window for servers that are in a maintenance window.

Has anyone already done this within the community?

I have found this in community posts https://community.grafana.com/t/how-to-specify-organisation-when-pausing-all-alerts/762 and was wondering if there was a way to do it via host?

My setup is telegraf agents on windows servers - pushing to Influxdb2.0 – Grafana visualisation and alerting via smtp.

Any help with this would be amazing and greatly appreciated

2

Hi,

Just to add to this -

" I have found this in community posts https://community.grafana.com/t/how-to-specify-organisation-when-pausing-all-alerts/762 and was wondering if there was a way to do it via host?"

If not would it be possible to do this via Dashboard ID?

Any help with this would be greatly received

3

Hello, ok i have the answer to this but struggling to get it to work correctly - this is the script that pulls the information based on dashboard tags:

#!/bin/bash
APIKEY=“MyAPIKey”
SERVER=“localhost”
PORT=“3000”
AlertsList=‘curl -H “Authorization: Bearer $APIKEY” -X GET http://$SERVER:$PORT/api/alerts?dashboardTag=prod 2>/dev/null | python -mjson.tool | grep ‘“id”’ | sed s/,//g | awk ‘{print $2}’ | sort -n’
for i in $AlertsList
do
curl -X POST http://$SERVER:$PORT/api/alerts/$i/pause -H “Content-type: application/json” -d ‘{“alertId”: ‘$i’, “paused”: true }’ -H “Authorization: Bearer $APIKEY”
echo “”
done

The issue I have is that when running the script I get:

“line 5: } | sort -n: command not found”

If I remove the | sort -n I get

line 5: } : command not found

Any ideas?

4

#!/bin/bash
APIKEY=“MyAPIKey”
SERVER=“localhost”
PORT=“3000”
AlertsList=curl -H "Authorization: Bearer $APIKEY" -X GET http://$SERVER:$PORT/api/alerts?dashboardTag=prod 2>/dev/null | python -m json.tool | grep ""id"" | sed s/,//g | awk '{print $2}' | sort -n
for i in $AlertsList
do
curl -X POST http://$SERVER:$PORT/api/alerts/$i/pause -H “Content-type: application/json” -d ‘{“alertId”: ‘$i’, “paused”: true }’ -H “Authorization: Bearer $APIKEY”
echo “”
done

The issue lay within the ` being ’ once changed this worked.

5

In case you find this useful I created a script to pause and resume alerts by ID. I believe you can put it in a cron job or a pipeline and manage schedules there. As a temporary solution

6

Hello thanks for that, the solution I needed was set in the script above where it looked for certain dashboard tags, then silenced them - I used the ‘at’ scheduler as this is better at setting certain dates and times as the times for silencing can be irregular each month

So basically would use the following to run the bash script

Type “at 11:00 PM 1/25/2021” (without the quotes)
Press return
Type in ./ then the filename example : ./prod-salert.sh
Press return
Pres CTRL + D to come out of it