Migration to unified alerting not working. No unified alerts are shown and no legacy alerts are retained either

Grafana Alerting
,
1

I’m interested in migrating from the legacy alerting system to unified alerting. According to the docs any version above 8.3 should do this automatically. I’m testing out an upgrade from 8.0.6 to 8.4.5.

Grafana comes up fine and I get no errors, all of my datasources and dashboards load however when I go to Alerting there are no rules, no contacts, no policies nothing is configured. If I go to view alerts in the old way under the panels on the dashboards there are no alerts there either.

There are some mentions of migrations in the log file but no errors. In particular this seems somewhat interesting:

 logger=migrator t=2022-04-05T18:01:36.59+0000 lvl=info msg="alerts found to migrate" alerts=0

And when I go to access the alert rules in the UI I see this in the logs:

logger=context t=2022-04-05T18:23:35.37+0000 lvl=info msg="Request Completed" method=GET path=/api/ruler/15/api/v1/rules status=404 remote_addr=172.18.251.97 time_ms=77 size=36 referer=https://grafana.dev-mgmt-eastus.bentleyhosted.com/alerting/list

I have also added the following to our config:

[unified_alerting]
enabled = true
2

what other config settings do you have set for alerting?

  • can you use this table to better understand how configuration options can interact with each other
3

The issue ended up being that we were not using persistent storage. We are using kubernetes containers which load their config and dashboards from files at startup. Unified alert migrations only run on existing entries in the database and our database was empty at the time of startup.

To test I enabled persistent storage and then upgraded from 8.0.6 to 8.4.5 and that did get me unified alerts.

That solves this problem but in turn brought up a new issue that I hadn’t thought of. We need the ability to load unified alerts from a config file, in the same way that we are able to do with configurations and dashboards. It sounds like Grafana has this on the roadmap but haven’t said yet when it would be in place.

2 Likes