Hi!
I experienced an issue when doing the upgrade from 12.3.4 to 12.4.1. Our installation is running on EC2 and deployment method is a rolling installation, so we have a mix of 12.3 and 12.4 during rollout.
As soon as the first 12.4 instance came online, I saw the following:
logger=storage.unified.migrations t=2026-03-18T09:47:46.850366791Z level=warn msg=“Failed to check if migration exists” migration=folders-dashboards error=“failed to check migration existence: Error 1146 (42S02): Table ‘grafana.unifiedstorage_migration_log’ doesn’t exist”
logger=ngalert.multiorg.alertmanager component=clustering t=2026-03-18T09:47:48.677155579Z level=warn received="unknown state key" len=33582 key=notificationlog:32
logger=live t=2026-03-18T09:51:19.286611407Z level=error msg="Error parsing channel" user=2273 client=0194f407-8c7b-4c1f-bffe-7410b11d24ae channel=32/grafana/dashboard/uid/iBYkQcC4z error="namespace does not reference a valid org ID: 32"
logger=live t=2026-03-18T09:51:19.328817743Z level=error msg="Error parsing channel" user=2232 client=7542013e-cb83-4399-8058-e6a552c20d3b channel=35/grafana/dashboard/uid/deilblmlb82yoctest error="namespace does not reference a valid org ID: 35"
The final 12.3 instance that was about to be terminated started to report encryption issues:
logger=secrets rule_uid=nA6hkx84z org_id=7 t=2026-03-18T09:58:09.79068897Z level=error msg=“Failed to decrypt secret” error=“failed to decrypt a secret encrypted with envelope encryption: envelope encryption is disabled”
logger=secrets.kvstore t=2026-03-18T09:58:09.790766408Z level=error msg=“error decrypting secret value” orgId=7 type=datasource namespace=“traffic-management” err=“failed to decrypt a secret encrypted with envelope encryption: envelope encryption is disabled”
logger=ngalert.scheduler rule_uid=nA6hkx84z org_id=7 version=10 fingerprint=12bdcfee10453167 now=2026-03-18T09:58:00Z rule_uid=nA6hkx84z org_id=7 t=2026-03-18T09:58:09.790878051Z level=error msg=“Failed to evaluate rule” attempt=2 max_attempts=3 next_attempt_in=1.63334242s error=“the result-set has errors that can be retried: failed to decrypt a secret encrypted with envelope encryption: envelope encryption is disabled”
logger=secrets rule_uid=z6ChzbUVk org_id=1 t=2026-03-18T09:58:09.794914635Z level=error msg=“Failed to decrypt secret” error=“failed to decrypt a secret encrypted with envelope encryption: envelope encryption is disabled”
I have envelope encryption disabled in both 12.3 and 12.4 so I am not sure how it decided to encrypt, furthermore there is no secret_key set in the config so not sure what it used to encrypt. This also meant rolling back was impossible as I didn’t know the encryption key so I had to restore the database from backup.
Any ideas? Annoying non-live upgraded successfully and I’ve recently tried a copy of our live DB on int and upgraded and that was successful too.