Database locked

Grafana
1

Today my email alerts have been going haywire and the Grafana webpage is not working well, it shows various server errors intermittently. Here are a few messages from the log:

logger=sqlstore.transactions t=2023-12-02T17:22:34.049093396Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked"
logger=authn.service t=2023-12-02T17:22:34.049160143Z level=error msg="Failed to authenticate request" client=auth.client.session error="[sqlstore.max-retries-reached] retry 1: database is locked"
logger=context userId=0 orgId=0 uname= t=2023-12-02T17:22:34.049209635Z level=error msg= error="[sqlstore.max-retries-reached] retry 1: database is locked" remote_addr=172.24.80.239 traceID=
logger=ngalert.scheduler t=2023-12-02T17:22:35.006682912Z level=error msg="Failed to update alert rules" error="[sqlstore.max-retries-reached] retry 1: database is locked"
logger=context userId=0 orgId=0 uname= t=2023-12-02T17:22:36.455593121Z level=error msg="Failed to get settings" error="[sqlstore.max-retries-reached] retry 1: database is locked"
logger=context userId=0 orgId=0 uname= t=2023-12-02T17:22:36.455780524Z level=error msg="Request Completed" method=GET path=/login status=500 remote_addr=172.24.80.239 time_ms=13140 duration=13.14025046s size=1349 referer= handler=/login

I’m running Grafana 10.2.0 in Docker. I’m not sure what caused this to happen, I’m not aware of any changes or crashing. How can I find the root cause?

2

I would use proper DB (MySQL, Postgresql) and not default SQLite. That may solve also your API live issues.

3

Interesting. I never thought much about the database because it’s so small. The bulk of my data lives in Prometheus or Loki. Looking online now, I see a lot of recommendations to switch to MySQL so I’ll give that a try. But I’m kind of surprised this is necessary because my grafana config is quite simple, just a few dashboards which are all stored as JSON files in the provisioning folder, and the config stored in environment variables.

4

Metric (traces, logs) data are not stored in Grafana DB. But dashboards (dashboard versions, alerts, users,…) are stored there. You may have a small DB (e. g. only one dashboard with one panel), but concurrent access to DB is weak point for SQLite.

1 Like
5

Ok I’ve made the switch to mariadb 11 and it’s looking good so far. Fingers crossed this solves it.
The live api issue is still present though.

1 Like