Unified Alerts fail to update with new data source after v8.5.3-v8.5.4 upgrade

Grafana Alerting
1

Created unified alerts in v8.5.3 that worked initially worked using Grafana v8.5.4. Then removed data source and alerts fire with error status. Recreated data sources using v8.5.4 with same name and tried assigning this to alert (or a different data source created in v8.5.3 or earlier). Consistently get a message of this form:

Failed to save rule: invalid rule specification at index [0]: failed to validate condition of alert rule <alert rule name>: invalid query A: data source not found: <data source id>

Requires recreating unified alerts from scratch. Issue appears similar to this:

3

@wkneewalden is this a typo or did you create alerts and then downgrade grafana to an earlier version? I started to open a bug report on your behalf because I don’t see anything like this reported in 8.5.x, but I wanted to confirm the scenario that got the database into this state.

4

I had unified (stand-alone) alerts created in v8.5.3, and then upgraded Grafana to v8.5.4, while keeping alerts. Those alerts referenced data sources that were created using v8.5.3 or earlier (likely an earlier version, but not sure which one). While using v8.5.4 I deleted the data sources, and then later recreated them (with same names that alerts reference) while using v8.5.4. The same alerts display error message listed above when attempting to reassign the data source for their query to new ones. From what I remember, it appeared to be an issue with stand alone alerts not being able to easily transfer their data source reference to new data source with same name.

5

They were Prometheus data sources by the way.

6

Observed a bug relating to slightly different behavior when upgrading from v8.5.4 to v8.5.11. Unified alerts fail to locate existing Prometheus data source. Had to create a new folder and then create replacement alerts in the new folder where it could detect the data source. Documented in a separate ticket.

7

I think that was fixed in v9, but if I recall correctly, one had to specify Prometheus as the default data source in order for it to appear. See toggle switch on the far right in screenshot below.

image
image1047×424 30.1 KB

1 Like
8

Appears that v9 fixed it, haven’t seen this happen with a few upgrades.