Best way to silence or disable alert notifications for database migration

Grafana MySQL
, ,
1

Grafana version: 8.4.3
OS: Linux Ubuntu 20.04.1 LTS
Self-hosted: true
Alerting: unified (some new some migrated)
Configs:

[alerting]
concurrent_render_limit      = 5
enabled                      = false
error_or_timeout             = alerting
evaluation_timeout_seconds   = 30
execute_alerts               = true
max_annotation_age           =
max_annotations_to_keep      = 0
max_attempts                 = 3
min_interval_seconds         = 1
nodata_or_nullvalues         = no_data
notification_timeout_seconds = 30

[unified_alerting]
admin_config_poll_interval        = 60s
alertmanager_config_poll_interval = 60s
disabled_orgs                     =
enabled                           = true
evaluation_timeout                = 30s
execute_alerts                    = true
ha_advertise_address              =
ha_gossip_interval                = 200ms
ha_listen_address                 = 0.0.0.0:9094
ha_peer_timeout                   = 15s
ha_peers                          =
ha_push_pull_interval             = 60s
max_attempts                      = 3
min_interval                      = 10s

  • What are you trying to achieve?
    I have a live Grafana system setup. I have been tasked with migrating the system to a new server.

  • How are you trying to achieve it?
    I setup the new system, then I did a mysqldump on the database for the live system.

  • What happened?
    I imported the dump file into the new database server. I started Grafana and it came up just fine. But it had not yet been connected to the datasources, so alerts fired and notifications got sent, which caused confusion among the teams in the company.

I looked through the database to see if there was an easy way to update all the Contact Points to use a temporary destination that would not bother everyone at the company. Thought I made a good change. Started Grafana again and still got notifications going to the usual contacts.

Having triggered false notifications twice now, I am gun shy about starting Grafana up again on the new system until I have figured out a way to ensure false notifications will not be sent again.

I am considering updating Contact Points on the live system and then doing another database dump to then re-import into the new system. And/or I could setup a silence for all the alerts before I dump the database.

Can anyone tell me if this will be adequate for keeping false notifications from being sent on the new system? Or is there a better way?

2

You can update the is_pause to true in the alert_rule table

1 Like