Issue with secrets after each restart

Hello,

in one of my “older” Grafana installations I have a strange secret related issue I don’t get fixed:

  • What Grafana version and what operating system are you using?
    v9.3.6 - Docker - MySQL DB

  • What are you trying to achieve?
    After a restart of the Grafana environment, it is not possible e.g. to add or change a datasource.

  • How are you trying to achieve it?

  • What happened?
    It shows only a error message in the UI.

  • What did you expect to happen?
    I’m able to add or change a datasource.

  • Can you copy/paste the configuration(s) that you are having problems with?

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

You can see this in the log:

logger=secrets t=2023-02-14T11:37:08.326180872Z level=error msg="Failed to get current data key" error="Error 1062: Duplicate entry '' for key 'PRIMARY'" label=2023-02-14/root@secretKey.v1
logger=secrets.kvstore t=2023-02-14T11:37:08.32623177Z level=error msg="error encrypting secret value" orgId=1 type=datasource namespace=Graphite err="Error 1062: Duplicate entry '' for key 'PRIMARY'"
logger=context userId=16 orgId=1 uname=USER_NAME t=2023-02-14T11:37:08.337803438Z level=error msg="Failed to add datasource" error="Error 1062: Duplicate entry '' for key 'PRIMARY'" remote_addr=CLIENT_IP traceID=
logger=context userId=16 orgId=1 uname=USER_NAME t=2023-02-14T11:37:08.337878078Z level=error msg="Request Completed" method=POST path=/api/datasources status=500 remote_addr=CLIENT_IP time_ms=151 duration=151.55471ms size=51 referer=https://GRAFANA_URL/datasources/new handler=/api/datasources/
  • Did you follow any online instructions? If so, what is the URL?

  • Workaround:

After trying around to “fix” the issue, I find this workaround:

bash-5.1$ grafana-cli admin secrets-migration rollback
INFO [02-14|11:41:08] Starting Grafana                         logger=settings version= commit= branch= compiled=1970-01-01T00:00:00Z
INFO [02-14|11:41:08] Config loaded from                       logger=settings file=/usr/share/grafana/conf/defaults.ini
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_DEFAULT_INSTANCE_NAME=Global-Grafana"
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_DEFAULT_FORCE_MIGRATION=true"
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_PATHS_DATA=/var/lib/grafana"
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_PATHS_LOGS=/var/log/grafana"
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_SERVER_ROOT_URL=https://GRAFANA_URL"
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_DATABASE_URL=mysql://grafana:xxxxx@DB_HOST:3306/grafana"
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_SECURITY_ADMIN_PASSWORD=*********"
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_AUTH_SIGV4_AUTH_ENABLED=true"
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_AUTH_ANONYMOUS_ORG_NAME=global"
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_AUTH_LDAP_ENABLED=true"
INFO [02-14|11:41:08] Path Home                                logger=settings path=/usr/share/grafana
INFO [02-14|11:41:08] Path Data                                logger=settings path=/var/lib/grafana
INFO [02-14|11:41:08] Path Logs                                logger=settings path=/var/log/grafana
INFO [02-14|11:41:08] Path Plugins                             logger=settings path=/var/lib/grafana/plugins
INFO [02-14|11:41:08] Path Provisioning                        logger=settings path=/etc/grafana/provisioning
INFO [02-14|11:41:08] App mode production                      logger=settings
INFO [02-14|11:41:08] Connecting to DB                         logger=sqlstore dbtype=mysql
INFO [02-14|11:41:08] Starting DB migrations                   logger=migrator
INFO [02-14|11:41:08] migrations completed                     logger=migrator performed=0 skipped=464 duration=754.02µs
INFO [02-14|11:41:08] Envelope encryption state                logger=secrets enabled=true current provider=secretKey.v1
INFO [02-14|11:41:08] Column dashboard_encrypted from dashboard_snapshot has been rolled back successfully logger=secrets.migrations
INFO [02-14|11:41:08] Column o_auth_access_token from user_auth has been rolled back successfully logger=secrets.migrations
INFO [02-14|11:41:08] Column o_auth_refresh_token from user_auth has been rolled back successfully logger=secrets.migrations
INFO [02-14|11:41:08] Column o_auth_token_type from user_auth has been rolled back successfully logger=secrets.migrations
INFO [02-14|11:41:08] Column value from secrets has been rolled back successfully logger=secrets.migrations
INFO [02-14|11:41:08] Secure json data secrets from data_source have been rolled back successfully logger=secrets.migrations
INFO [02-14|11:41:08] Secure json data secrets from plugin_setting have been rolled back successfully logger=secrets.migrations
INFO [02-14|11:41:08] Alerting configuration secrets have been rolled back successfully logger=secrets.migrations

bash-5.1$ grafana-cli admin secrets-migration re-encrypt
INFO [02-14|11:41:11] Starting Grafana                         logger=settings version= commit= branch= compiled=1970-01-01T00:00:00Z
INFO [02-14|11:41:11] Config loaded from                       logger=settings file=/usr/share/grafana/conf/defaults.ini
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_DEFAULT_INSTANCE_NAME=Global-Grafana"
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_DEFAULT_FORCE_MIGRATION=true"
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_PATHS_DATA=/var/lib/grafana"
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_PATHS_LOGS=/var/log/grafana"
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_SERVER_ROOT_URL=https://GRAFANA_URL"
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_DATABASE_URL=mysql://grafana:xxxxx@DB_HOST:3306/grafana"
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_SECURITY_ADMIN_PASSWORD=*********"
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_AUTH_SIGV4_AUTH_ENABLED=true"
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_AUTH_ANONYMOUS_ORG_NAME=global"
INFO [02-14|11:39:36] Config overridden from Environment variable logger=settings var="GF_AUTH_LDAP_ENABLED=true"
INFO [02-14|11:41:11] Path Home                                logger=settings path=/usr/share/grafana
INFO [02-14|11:41:11] Path Data                                logger=settings path=/var/lib/grafana
INFO [02-14|11:41:11] Path Logs                                logger=settings path=/var/log/grafana
INFO [02-14|11:41:11] Path Plugins                             logger=settings path=/var/lib/grafana/plugins
INFO [02-14|11:41:11] Path Provisioning                        logger=settings path=/etc/grafana/provisioning
INFO [02-14|11:41:11] App mode production                      logger=settings
INFO [02-14|11:41:11] Connecting to DB                         logger=sqlstore dbtype=mysql
INFO [02-14|11:41:11] Starting DB migrations                   logger=migrator
INFO [02-14|11:41:11] migrations completed                     logger=migrator performed=0 skipped=464 duration=2.210148ms
INFO [02-14|11:41:11] Envelope encryption state                logger=secrets enabled=true current provider=secretKey.v1
INFO [02-14|11:41:11] Column dashboard_encrypted from dashboard_snapshot has been re-encrypted successfully logger=secrets.migrations
INFO [02-14|11:41:11] Column o_auth_access_token from user_auth has been re-encrypted successfully logger=secrets.migrations
INFO [02-14|11:41:11] Column o_auth_refresh_token from user_auth has been re-encrypted successfully logger=secrets.migrations
INFO [02-14|11:41:11] Column o_auth_token_type from user_auth has been re-encrypted successfully logger=secrets.migrations
INFO [02-14|11:41:11] Column value from secrets has been re-encrypted successfully logger=secrets.migrations
INFO [02-14|11:41:11] Secure json data secrets from data_source have been re-encrypted successfully logger=secrets.migrations
INFO [02-14|11:41:11] Secure json data secrets from plugin_setting have been re-encrypted successfully logger=secrets.migrations
INFO [02-14|11:41:11] Alerting configuration secrets have been re-encrypted successfully logger=secrets.migrations

After this two steps every error disappear and I’m able to add / change a datasource again.
Until the next reboot…

Is someone able to help me here?

Thanks a lot,
Alex