Error getting secret value

Grafana Prometheus
1
  • What Grafana version and what operating system are you using?
    11.4.0
  • What are you trying to achieve?
    Make my datasource work
  • How are you trying to achieve it?
    I have my grafana and prom in a docker env. My grafana has provisionned datasources, dashboards and provisionned grafana.ini.
  • What happened?
    Everything was working fine for the 6 past months. And since few days, i began to have those errors :
    logger=secrets.kvstore t=2025-02-07T13:49:57.689111725Z level=error msg=“error getting secret value” orgId=1 type=datasource namespace=prometheus-bla err=“context canceled”
    logger=infra.usagestats.collector t=2025-02-07T13:49:57.689132327Z level=error msg=“Failed to get HTTP client for Prometheus data source” error=“context canceled”. When it occurs, on y dashboards, the panel with the data linked to the query is in failure as the datasource is not reachable.
  • What did you expect to happen?
    I’d like to have my grafana working as it was for the past months.
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    logger=secrets.kvstore t=2025-02-07T13:49:57.689111725Z level=error msg=“error getting secret value” orgId=1 type=datasource namespace=prometheus-bla err=“context canceled”
    logger=infra.usagestats.collector t=2025-02-07T13:49:57.689132327Z level=error msg=“Failed to get HTTP client for Prometheus data source” error=“context canceled”
  • Did you follow any online instructions? If so, what is the URL?
  • yes, i found several topics related to this error : Cloudwatch Datasource NoData -> error getting secret value · Issue #91122 · grafana/grafana · GitHub : i changed the timeout in grafana.ini (from 30s to 60s) and still have the error
  • Spike in memory usage overnight causing crash · Issue #91496 · grafana/grafana · GitHub : checked memory on my server, was not 100%, resized my server just to be sure, it did not fixed my issue
2

Hi,

It’s a long shot but I found one place in codebase that could create such an error. Do you use database? Could it be that the connection to the database is unstable or something like this?

3

Hi, sorry for the delay, i don’t use database. Here is the docker code :
grafana:
image: grafana/grafana-enterprise
container_name: grafana
restart: on-failure
ports:
- ‘3000:3000’
volumes:
- ./etc/grafana/config/custom.ini:/etc/grafana/grafana.ini:rw
- ./etc/grafana/provisioning/:/etc/grafana/provisioning/
- ./etc/grafana/ldap.toml:/etc/grafana/ldap.toml
env_file: “.env”
environment:
GF_USERS_ALLOW_SIGN_UP: “false”
GF_AUTH_LDAP_ENABLED: “true”
GF_AUTH_LDAP_CONFIG_FILE: /etc/grafana/ldap.toml

Nothing fancy on the grafana.ini, just set the admin password on start.