Low performance after migration to the unified alerting

Grafana Alerting
,
1

  • What Grafana version and what operating system are you using? - 8.4.6

  • What are you trying to achieve? - The same performance with a legacy alerting.

  • What happened? - After migration to the unified alerting Grafana working very slow.
    Grafana configuration:
    Organizations count - 28
    Alerts count - 6517
    Dashboards count - 2420
    Notification channel count - 2110

My environment:

  • Grafana version 8.4.6

  • Grafana is self-hosted

  • I used a Unified Alerting

  • These alerts in question migrated from the legacy platform into Unified Alerting.

  • Grafana alets configuration
    #################################### Alerting ############################
    [unified_alerting]
    enabled = true
    admin_config_poll_interval = 60s
    alertmanager_config_poll_interval = 60s
    ha_listen_address = “0.0.0.0:9094”
    ha_advertise_address = “”
    ha_peers = “”
    ha_peer_timeout = 15s
    ha_gossip_interval = 200ms
    ha_push_pull_interval = 60s
    execute_alerts = true
    evaluation_timeout = 30s
    max_attempts = 3
    min_interval = 10s

    [alerting]
    enabled = false
    execute_alerts = true

  • List the datasource associated with the alert:
    InfluxDB

2

welcome to the :grafana: community @hryhoriididenko!

Can you share more about the performance differences you’re seeing?

  • is there an example of a task that has slowed? how long does it take now and how long did it take before?
  • did you do a side-by-side upgrade where you can compare the two instances?
  • were any other changes made when you enabled unified alerting - for example, new server, networking, or OS?
3
  • is there an example of a task that has slowed? how long does it take now and how long did it take before? - all tasks, for example, open list of orgs, open list of users, open list of alerts, open dashboard. And previously on legacy alerts, Grafana consumes a 1CPU and 1GB RAM, on a unified alert consumes 6CPU 7GB RAM. A server where we run a grafana has 16 CPUs and 64 GB RAM.
  • did you do a side-by-side upgrade where you can compare the two instances? - yes
  • were any other changes made when you enabled unified alerting - for example, new server, networking, or OS? - no
4

Had the same issue. Upgrading from Grafana 8.2.3 to 9.3.6