Hello,
Last week me decided to migrate to a new instance of Grafana because we had issues with DatasourceError messages being sent all the time. We did not migrate any data, but started from a fresh instance
The problem is that we are still receiving these DatasourceError messsages, even being on PostgreSQL and high availability setup. I also don’t see any error message in the logs, just standard messages that the notification has been sent. E.g. below during this period we received 4 DataSourceError alerts, but nothing specific on the logs:
logger=ngalert.sender.router rule_uid=f7212f42-e845-48c6-b30b-d34723fadadf org_id=1 t=2024-03-02T04:34:00.59207316Z level=info msg="Sending alerts to local notifier" count=1
logger=cleanup t=2024-03-02T04:33:47.455662515Z level=info msg="Completed cleanup jobs" duration=420.145918ms
logger=plugins.update.checker t=2024-03-02T04:33:47.418743418Z level=info msg="Update check succeeded" duration=38.955657ms
logger=grafana.update.checker t=2024-03-02T04:33:47.182562455Z level=info msg="Update check succeeded" duration=51.999806ms
logger=cleanup t=2024-03-02T04:33:33.950559462Z level=info msg="Completed cleanup jobs" duration=401.291199ms
logger=grafana.update.checker t=2024-03-02T04:33:33.735925505Z level=info msg="Update check succeeded" duration=80.058385ms
logger=plugins.update.checker t=2024-03-02T04:33:33.69458702Z level=info msg="Update check succeeded" duration=58.154509ms
logger=ngalert.sender.router rule_uid=fb102ad4-87d4-4c30-af58-496b3c23955e org_id=1 t=2024-03-02T04:33:09.90225472Z level=info msg="Sending alerts to local notifier" count=1
logger=ngalert.sender.router rule_uid=f7212f42-e845-48c6-b30b-d34723fadadf org_id=1 t=2024-03-02T04:33:09.399955974Z level=info msg="Sending alerts to local notifier" count=1
logger=ngalert.sender.router rule_uid=f7212f42-e845-48c6-b30b-d34723fadadf org_id=1 t=2024-03-02T04:33:07.069167528Z level=info msg="Sending alerts to local notifier" count=1
logger=ngalert.sender.router rule_uid=fb102ad4-87d4-4c30-af58-496b3c23955e org_id=1 t=2024-03-02T04:33:03.071261649Z level=info msg="Sending alerts to local notifier" count=1
logger=ngalert.sender.router rule_uid=f7212f42-e845-48c6-b30b-d34723fadadf org_id=1 t=2024-03-02T04:32:09.930490227Z level=info msg="Sending alerts to local notifier" count=1
logger=ngalert.sender.router rule_uid=fb102ad4-87d4-4c30-af58-496b3c23955e org_id=1 t=2024-03-02T04:32:05.931884622Z level=info msg="Sending alerts to local notifier" count=1
logger=ngalert.sender.router rule_uid=fb102ad4-87d4-4c30-af58-496b3c23955e org_id=1 t=2024-03-02T04:32:04.164316086Z level=info msg="Sending alerts to local notifier" count=1
logger=ngalert.sender.router rule_uid=f7212f42-e845-48c6-b30b-d34723fadadf org_id=1 t=2024-03-02T04:32:03.664795715Z level=info msg="Sending alerts to local notifier" count=1
logger=ngalert.sender.router rule_uid=f7212f42-e845-48c6-b30b-d34723fadadf org_id=1 t=2024-03-02T04:31:09.281924046Z level=info msg="Sending alerts to local notifier" count=1
Any idea how to definitely get rid of these DatasourceError ?