Hi. Every morning at 6 am I get a warning notification due to “no value”. The reason for the error appears as follows. It fixes in a few minutes, but it causes me to receive dozens of notifications. How can I solve this problem?
Version: Grafana v9.5.1 (bc353e4b2d)
Error: failed to build query ‘A’: [sqlstore.max-retries-reached] retry 1: database is locked
I’m also experiencing a lot of those after upgrade from 9.4.7 to 9.5.2
Hello @here,
Please see this GitHub link which provided a good workaround.
Also, I believe that you are using SQLIte as Grafana backend DB so best would be to switch to MySQL or PostgreSQL (same link also has the information).
I hope this helps.
Thanks for the hint, I enabled WAL for sqlite3 using GF_DATABASE_WAL=true env variable (running dockerized grafana).
See also enabling wal = true in grafana.ini as default for Grafana SQLite installation · Issue #65115 · grafana/grafana · GitHub (about enabling it by default).
Update Status
It seems that the Engineering team has made some changes in the code to improve the Database stability and avoid such locks.
Still, the best and suggested method is to switch to either MySQL or PostgreSQL.
Therefore marking this post as resolved
We are using the ubuntu OS, grafana with proemetheus datasource i’m using, we are also getting this type of flase alerts datasource:locked.
How to resolved this issue?
I also got this error before. solved it by changing Grafana db to my sql. (default db was sqlite btw)
Hi @viswa8pk and @haileymoon
From what I can tell, there are almost no issues reported by users especially on the GitHub issues about this problem as after the PR it was significantly improved to work better with the default SQLite backend Database.
Can you please tell us what Grafana version you are using and on which environment you running it e.g. Docker, Kubernetes, local etc. ?
we are using the grafana 10.2.3 and prometheus 2.49.0 using this setup in my local linux machines, we are using 1 yearly but this false mail’s getting now a days.
refer this image… we are getting this type of false mails on the midnight.
Could you please change the default log level to debug mode?
To do this go to the server where grafana is installed and edit the configuration file (/etc/grafana/grafana.ini)
There you will this section;
#################################### Logging ##########################
[log]
# Either "console", "file", "syslog". Default is console and file
# Use space to separate multiple modes, e.g. "console file"
;mode = console file
# Either "debug", "info", "warn", "error", "critical", default is "info"
;level = info
Set the parameter level to debug and remove the colon in the beginning i.e.
level = debug
Then restart the grafana service and resent the logs once the issue occurs again.
we are changed the
level = debug in grafana.ini, after that today morning again we got the false mails Error failed to build query ‘A’: [sqlstore.max-retries-reached] retry 1: database is locked
This is the log on the same time in my grafana logs, kindly refer it
logger=ngalert.state.manager rule_uid=c8859c7e-2206-49d1-ba55-9519a8f1b025 org_id=1 instance="__name__=probe_success, instance=http://192.168.1.166:9308, job=blackbox_exporter" t=2024-01-10T04:03:31.039151285+05:30 level=debug msg="Keeping state" state=Alerting previous_ends_at=2024-01-10T04:05:20+05:30 next_ends_at=2024-01-10T04:05:30+05:30
logger=ngalert.notifier.alertmanager 1=(MISSING) t=2024-01-10T04:03:31.044652566+05:30 level=debug component=alertmanager orgID=1 msg="Putting alert" alert="Kafka Exporter 166[a391688][active]" starts_at=2024-01-10T02:38:50+05:30 ends_at=2024-01-10T04:05:30+05:30
logger=ngalert.state.manager rule_uid=b3f0e71d-259f-4d3e-b0c4-cf15246e3a1a org_id=1 instance="__name__=probe_success, instance=http://192.168.1.166:4041, job=blackbox_exporter" t=2024-01-10T04:03:33.089615902+05:30 level=debug msg="Keeping state" state=Alerting previous_ends_at=2024-01-10T04:05:20+05:30 next_ends_at=2024-01-10T04:05:30+05:30
logger=ngalert.state.manager rule_uid=ca214062-704a-49c0-84fd-ae9b6d9f36d2 org_id=1 instance="__name__=probe_success, instance=http://192.168.1.166:4042, job=blackbox_exporter" t=2024-01-10T04:03:33.602731835+05:30 level=debug msg="Keeping state" state=Alerting previous_ends_at=2024-01-10T04:05:20+05:30 next_ends_at=2024-01-10T04:05:30+05:30
logger=ngalert.state.manager rule_uid=d80af3ba-807f-4225-9edc-6360714fb783 org_id=1 instance="__name__=probe_success, instance=http://192.168.1.166:4040, job=blackbox_exporter" t=2024-01-10T04:03:34.114786528+05:30 level=debug msg="Keeping state" state=Alerting previous_ends_at=2024-01-10T04:05:20+05:30 next_ends_at=2024-01-10T04:05:30+05:30
logger=ngalert.state.manager rule_uid=f55ddd12-0516-41d1-8652-79030676b0b5 org_id=1 instance="__name__=probe_success, instance=http://192.168.1.166:4042, job=blackbox_exporter" t=2024-01-10T04:03:37.705071461+05:30 level=debug msg="Keeping state" state=Alerting previous_ends_at=2024-01-10T04:05:20+05:30 next_ends_at=2024-01-10T04:05:30+05:30
logger=ngalert.state.manager rule_uid=f55ddd12-0516-41d1-8652-79030676b0b5 org_id=1 instance="__name__=probe_success, instance=http://192.168.1.166:4042, job=blackbox_exporter" t=2024-01-10T04:03:42.32031393+05:30 level=debug msg="Keeping state" state=Alerting previous_ends_at=2024-01-10T04:05:30+05:30 next_ends_at=2024-01-10T04:05:40+05:30
logger=ngalert.state.manager rule_uid=c8859c7e-2206-49d1-ba55-9519a8f1b025 org_id=1 instance="__name__=probe_success, instance=http://192.168.1.166:9308, job=blackbox_exporter" t=2024-01-10T04:03:45.652023262+05:30 level=debug msg="Keeping state" state=Alerting previous_ends_at=2024-01-10T04:05:30+05:30 next_ends_at=2024-01-10T04:05:40+05:30
logger=ngalert.state.manager rule_uid=b3f0e71d-259f-4d3e-b0c4-cf15246e3a1a org_id=1 instance="__name__=probe_success, instance=http://192.168.1.166:4041, job=blackbox_exporter" t=2024-01-10T04:03:47.702953373+05:30 level=debug msg="Keeping state" state=Alerting previous_ends_at=2024-01-10T04:05:30+05:30 next_ends_at=2024-01-10T04:05:40+05:30
logger=ngalert.state.manager rule_uid=ca214062-704a-49c0-84fd-ae9b6d9f36d2 org_id=1 instance="__name__=probe_success, instance=http://192.168.1.166:4042, job=blackbox_exporter" t=2024-01-10T04:03:48.216000227+05:30 level=debug msg="Keeping state" state=Alerting previous_ends_at=2024-01-10T04:05:30+05:30 next_ends_at=2024-01-10T04:05:40+05:30
logger=ngalert.state.manager rule_uid=d80af3ba-807f-4225-9edc-6360714fb783 org_id=1 instance="__name__=probe_success, instance=http://192.168.1.166:4040, job=blackbox_exporter" t=2024-01-10T04:03:48.730774336+05:30 level=debug msg="Keeping state" state=Alerting previous_ends_at=2024-01-10T04:05:30+05:30 next_ends_at=2024-01-10T04:05:40+05:30
logger=ngalert.notifier.alertmanager 1=(MISSING) t=2024-01-10T04:05:30.279372155+05:30 level=debug component=alertmanager orgID=1 component=dispatcher aggrGroup="{}/{team=\"- it\"}:{alertname=\"203 Techexel RAM \", grafana_folder=\"Windows Server Alerts\"}" msg=flushing alerts="[203 Techexel RAM [01ce6f3][active]]"
logger=ngalert.state.manager rule_uid=c8859c7e-2206-49d1-ba55-9519a8f1b025 org_id=1 instance= t=2024-01-10T04:05:40.194077197+05:30 level=debug msg="Changing state" previous_state=Normal next_state=Error previous_ends_at=2024-01-10T04:05:30+05:30 next_ends_at=2024-01-10T04:07:30+05:30
logger=ngalert.state.manager rule_uid=bc0c1576-4669-4a45-9f17-c9a6a462369b org_id=1 instance= t=2024-01-10T04:05:40.449560145+05:30 level=debug msg="Changing state" previous_state=Normal next_state=Error previous_ends_at=2024-01-10T04:05:30+05:30 next_ends_at=2024-01-10T04:07:30+05:30
logger=ngalert.state.manager rule_uid=a5f80a93-6000-4f02-9d15-f953b8bc33c1 org_id=1 instance= t=2024-01-10T04:05:40.706679573+05:30 level=debug msg="Changing state" previous_state=Normal next_state=Error previous_ends_at=2024-01-10T04:05:30+05:30 next_ends_at=2024-01-10T04:07:30+05:30
logger=ngalert.state.manager rule_uid=f543c5cb-5f5e-4880-a54d-cabbf6b044ab org_id=1 instance= t=2024-01-10T04:05:40.962770004+05:30 level=debug msg="Changing state" previous_state=Normal next_state=Error previous_ends_at=2024-01-10T04:05:30+05:30 next_ends_at=2024-01-10T04:07:30+05:30
logger=ngalert.notifier.alertmanager 1=(MISSING) t=2024-01-10T04:05:41.649843474+05:30 level=debug component=alertmanager orgID=1 msg="Putting alert" alert=DatasourceError[305dd1d][active] starts_at=2024-01-10T04:05:30+05:30 ends_at=2024-01-10T04:07:30+05:30
logger=ngalert.notifier.alertmanager 1=(MISSING) t=2024-01-10T04:05:41.815364371+05:30 level=debug component=alertmanager orgID=1 msg="Putting alert" alert=DatasourceError[d68547e][active] starts_at=2024-01-10T04:05:30+05:30 ends_at=2024-01-10T04:07:30+05:30
logger=ngalert.notifier.alertmanager 1=(MISSING) t=2024-01-10T04:05:42.043493465+05:30 level=debug component=alertmanager orgID=1 msg="Putting alert" alert=DatasourceError[d1a0459][active] starts_at=2024-01-10T04:05:30+05:30 ends_at=2024-01-10T04:07:30+05:30
logger=ngalert.notifier.alertmanager 1=(MISSING) t=2024-01-10T04:05:42.151338353+05:30 level=debug component=alertmanager orgID=1 msg="Putting alert" alert=DatasourceError[71ab0da][active] starts_at=2024-01-10T04:05:30+05:30 ends_at=2024-01-10T04:07:30+05:30
logger=ngalert.state.manager rule_uid=b3f0e71d-259f-4d3e-b0c4-cf15246e3a1a org_id=1 instance= t=2024-01-10T04:05:42.243824022+05:30 level=debug msg="Changing state" previous_state=Normal next_state=Error previous_ends_at=2024-01-10T04:05:30+05:30 next_ends_at=2024-01-10T04:07:30+05:30
logger=ngalert.notifier.alertmanager 1=(MISSING) t=2024-01-10T04:05:42.331357478+05:30 level=debug component=alertmanager orgID=1 msg="Putting alert" alert=DatasourceError[465515d][active] starts_at=2024-01-10T04:05:30+05:30 ends_at=2024-01-10T04:07:30+05:30
logger=ngalert.notifier.alertmanager 1=(MISSING) t=2024-01-10T04:05:42.614304517+05:30 level=debug component=alertmanager orgID=1 msg="Putting alert" alert=DatasourceError[bee9bd0][active] starts_at=2024-01-10T04:05:30+05:30 ends_at=2024-01-10T04:07:30+05:30
logger=ngalert.notifier.alertmanager 1=(MISSING) t=2024-01-10T04:05:42.994733279+05:30 level=debug component=alertmanager orgID=1 msg="Putting alert" alert=DatasourceError[ceadd91][active] starts_at=2024-01-10T04:05:30+05:30 ends_at=2024-01-10T04:07:30+05:30
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=f55ddd12-0516-41d1-8652-79030676b0b5 org_id=1 t=2024-01-10T04:06:30.27122652+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:06:30+05:30 step=15s query="probe_success{instance=~\"http://192\\\\.168\\\\.1\\\\.166:4042\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=cca72ba3-a805-4731-baba-2b387eb11554 org_id=1 t=2024-01-10T04:06:31.29706964+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:06:30+05:30 step=15s query="probe_success{instance=~\"http://192\\\\.168\\\\.1\\\\.164:4041\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=fe028336-5c61-45db-bd72-a4ed7d6e1504 org_id=1 t=2024-01-10T04:06:32.065385902+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:06:30+05:30 step=15s query="kafka_topic_partitions{instance=\"192.168.1.163:9308\",job=\"Kafka\",topic=\"__consumer_offsets\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=d324c6e9-af5a-4784-887d-e5d7fd5f2b5c org_id=1 t=2024-01-10T04:06:32.580441827+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:06:30+05:30 step=15s query="node_systemd_unit_state{instance=\"192.168.1.226:9100\",name=\"tomcat.service\",state=\"active\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=fc7a25c0-ac7e-49dc-98e5-a561ec6659ce org_id=1 t=2024-01-10T04:06:33.347499021+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:06:30+05:30 step=15s query="probe_success{instance=~\"http://192\\\\.168\\\\.1\\\\.164:4043\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=c8859c7e-2206-49d1-ba55-9519a8f1b025 org_id=1 t=2024-01-10T04:06:33.604463261+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:06:30+05:30 step=15s query="probe_success{instance=~\"http://192\\\\.168\\\\.1\\\\.166:9308\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=e8ff21f5-0ad6-42af-932d-c6f42cba34d0 org_id=1 t=2024-01-10T04:06:34.630230969+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:06:30+05:30 step=15s query="windows_netframework_clrloading_appdomains_loaded_total{instance=\"192.168.1.224:9115\",process=\"Broadcast_NSE\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=b77861fe-bf72-424c-a9cc-ffe2e13b6c2b org_id=1 t=2024-01-10T04:06:34.886365526+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:06:30+05:30 step=15s query="windows_netframework_clrloading_appdomains_loaded_total{instance=\"192.168.1.224:9115\",process=\"Broadcast_NFO\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=a1603afd-85b7-404b-a2b2-a0581b3c9bec org_id=1 t=2024-01-10T04:06:35.398745355+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:06:30+05:30 step=15s query="probe_success{instance=~\"http://192\\\\.168\\\\.1\\\\.164:4040\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=b3f0e71d-259f-4d3e-b0c4-cf15246e3a1a org_id=1 t=2024-01-10T04:06:35.911669923+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:06:30+05:30 step=15s query="probe_success{instance=~\"http://192\\\\.168\\\\.1\\\\.166:4041\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=ca214062-704a-49c0-84fd-ae9b6d9f36d2 org_id=1 t=2024-01-10T04:06:36.168792881+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:06:30+05:30 step=15s query="probe_success{instance=~\"http://192\\\\.168\\\\.1\\\\.166:4042\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=df40d1ad-a485-4750-9132-4028ccd42b25 org_id=1 t=2024-01-10T04:06:36.425500949+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:06:30+05:30 step=15s query="probe_success{instance=~\"http://192\\\\.168\\\\.1\\\\.164:4042\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=d80af3ba-807f-4225-9edc-6360714fb783 org_id=1 t=2024-01-10T04:06:36.681721296+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:06:30+05:30 step=15s query="probe_success{instance=~\"http://192\\\\.168\\\\.1\\\\.166:4040\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=dd81e13e-2655-46aa-938b-0506d73bd9e6 org_id=1 t=2024-01-10T04:06:37.450182539+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:06:30+05:30 step=15s query="kafka_topic_partitions{instance=\"192.168.1.163:9308\",job=\"Kafka\",topic=\"_schemas\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=f813b328-3a60-4d76-8d05-2ed77a43276c org_id=1 t=2024-01-10T04:06:37.964064806+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:06:30+05:30 step=15s query="probe_success{instance=~\"http://192\\\\.168\\\\.1\\\\.165:9308\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=fba371e0-d253-4aa0-b1fa-ad9628777740 org_id=1 t=2024-01-10T04:06:39.758255509+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:06:30+05:30 step=15s query="windows_netframework_clrloading_appdomains_loaded_total{instance=\"192.168.1.224:9115\",process=\"Broadcast_MCX\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=ebfe6394-35d1-44a9-8a56-c5630c212908 org_id=1 t=2024-01-10T04:07:35.13986982+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:07:30+05:30 step=15s query="(sum(windows_logical_disk_size_bytes{volume!~\"Harddisk.*\", instance=\"192.168.1.87:9115\"}) by (instance) - sum(windows_logical_disk_free_bytes{volume!~\"Harddisk.*\", instance=\"192.168.1.87:9115\"}) by (instance)) / sum(windows_logical_disk_size_bytes{volume!~\"Harddisk.*\", instance=\"192.168.1.87:9115\"}) by (instance) * 100"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=d8e086b4-ad81-488e-8fdd-58e79c1b6232 org_id=1 t=2024-01-10T04:07:35.394818503+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:07:30+05:30 step=15s query="sum(delta(kafka_topic_partition_current_offset{instance=~'192.168.1.164:9308', topic=~\"(_schemas|enrich\\\\.datastream\\\\.mcx|enrich\\\\.datastream\\\\.nfo|enrich\\\\.datastream\\\\.nse|enrich\\\\.datastream\\\\.raw)\"}[5m])/5) by (topic)"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=d5087713-1138-460d-9756-2f98db606b14 org_id=1 t=2024-01-10T04:07:35.9080562+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:07:30+05:30 step=15s query="(windows_cs_physical_memory_bytes{instance=~\"192.168.1.87:9115\"} - windows_os_physical_memory_free_bytes{instance=~\"192.168.1.87:9115\"}) / windows_cs_physical_memory_bytes{instance=~\"192.168.1.87:9115\"} * 100"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=cd2c35a7-e122-40bd-94e4-1539693cf973 org_id=1 t=2024-01-10T04:07:36.933498685+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:07:30+05:30 step=15s query="100 - (avg by (instance) (irate(windows_cpu_time_total{mode=\"idle\", instance=~\"192.168.1.87:9115\"}[1m])) * 100)"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=b9cece62-e124-4266-aa84-a091cc1173eb org_id=1 t=2024-01-10T04:07:38.728124415+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:07:30+05:30 step=15s query="sum(delta(kafka_topic_partition_current_offset{instance=~'192.168.1.163:9308', topic=~\"(_schemas|enrich\\\\.datastream\\\\.mcx|enrich\\\\.datastream\\\\.nfo|enrich\\\\.datastream\\\\.nse|enrich\\\\.datastream\\\\.raw)\"}[5m])/5) by (topic)"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=d3e77a02-ec7d-41dc-ba00-c21049cb252b org_id=1 t=2024-01-10T04:08:44.761138438+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:08:30+05:30 step=15s query="windows_service_state{instance=\"192.168.1.204:9115\",name=\"enrichaccordpostgresdatasync\",state=\"running\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=ddd20cd4-252d-4358-b2e3-02d93ffe557f org_id=1 t=2024-01-10T04:08:45.018492692+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:08:30+05:30 step=15s query="windows_service_state{instance=\"192.168.1.204:9115\",name=\"leadsquarservices\",state=\"running\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=c06364e0-9bc1-4d4d-96e2-86eb806f7151 org_id=1 t=2024-01-10T04:08:45.530352135+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:08:30+05:30 step=15s query="node_systemd_unit_state{instance=\"192.168.1.121:9100\",name=\"tomcat.service\",state=\"active\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=c1e7f80e-71b6-403e-8c03-4823cd878555 org_id=1 t=2024-01-10T04:08:45.787216386+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:08:30+05:30 step=15s query="node_systemd_unit_state{instance=\"192.168.1.194:9100\",name=\"postgresql@14-main.service\",state=\"active\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=c10b2a5d-98b0-4084-a507-04627abde885 org_id=1 t=2024-01-10T04:08:46.920269427+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:08:30+05:30 step=15s query="windows_service_state{instance=\"192.168.1.204:9115\",name=\"enrichorcaproservice\",state=\"running\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=a5f80a93-6000-4f02-9d15-f953b8bc33c1 org_id=1 t=2024-01-10T04:08:48.758995784+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:08:30+05:30 step=15s query="node_systemd_unit_state{instance=\"192.168.1.225:9100\",name=\"postgresql-14.service\",state=\"active\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=a29e13eb-98e5-487f-ad14-1601730d82ed org_id=1 t=2024-01-10T04:10:30.0103427+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:10:30+05:30 step=15s query="windows_process_handles{instance=\"192.168.1.186:9115\",process=\"redis-server\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=f0f9d7c6-8aed-4676-b153-49277c0b97c1 org_id=1 t=2024-01-10T04:10:30.267438107+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:10:30+05:30 step=15s query="node_time_seconds{instance=\"192.168.1.134:9100\"} - node_boot_time_seconds{instance=\"192.168.1.134:9100\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=bc0c1576-4669-4a45-9f17-c9a6a462369b org_id=1 t=2024-01-10T04:10:35.139024734+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:10:30+05:30 step=15s query="node_time_seconds{instance=\"192.168.1.132:9100\"} - node_boot_time_seconds{instance=\"192.168.1.132:9100\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=d66e4e94-f45f-4adb-a754-db817fe424c2 org_id=1 t=2024-01-10T04:10:38.984751793+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:10:30+05:30 step=15s query="node_time_seconds{instance=\"192.168.1.136:9100\"} - node_boot_time_seconds{instance=\"192.168.1.136:9100\"}"
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=f7b986d6-07ee-43df-9492-f253439029fe org_id=1 t=2024-01-10T04:10:39.754948724+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:10:30+05:30 step=15s query="windows_os_time{instance=~\"192.168.1.87:9115\"} - on () windows_process_start_time{creating_process_id=\"0\",instance=\"192.168.1.87:9115\",process=\"Idle\",process_id=\"0\"}"
Thanks for the logs, but I do not see any messages in there which says “database is locked”.
Instead I see some other errors:
logger=ngalert.notifier.alertmanager 1=(MISSING) t=2024-01-10T04:05:30.279372155+05:30 level=debug component=alertmanager orgID=1 component=dispatcher aggrGroup="{}/{team=\"- it\"}:{alertname=\"203 Techexel RAM \", grafana_folder=\"Windows Server Alerts\"}" msg=flushing alerts="[203 Techexel RAM [01ce6f3][active]]"
logger=ngalert.state.manager rule_uid=c8859c7e-2206-49d1-ba55-9519a8f1b025 org_id=1 instance= t=2024-01-10T04:05:40.194077197+05:30 level=debug msg="Changing state" previous_state=Normal next_state=Error previous_ends_at=2024-01-10T04:05:30+05:30 next_ends_at=2024-01-10T04:07:30+05:30
logger=ngalert.state.manager rule_uid=bc0c1576-4669-4a45-9f17-c9a6a462369b org_id=1 instance= t=2024-01-10T04:05:40.449560145+05:30 level=debug msg="Changing state" previous_state=Normal next_state=Error previous_ends_at=2024-01-10T04:05:30+05:30 next_ends_at=2024-01-10T04:07:30+05:30
logger=ngalert.state.manager rule_uid=a5f80a93-6000-4f02-9d15-f953b8bc33c1 org_id=1 instance= t=2024-01-10T04:05:40.706679573+05:30 level=debug msg="Changing state" previous_state=Normal next_state=Error previous_ends_at=2024-01-10T04:05:30+05:30 next_ends_at=2024-01-10T04:07:30+05:30
logger=ngalert.state.manager rule_uid=f543c5cb-5f5e-4880-a54d-cabbf6b044ab org_id=1 instance= t=2024-01-10T04:05:40.962770004+05:30 level=debug msg="Changing state" previous_state=Normal next_state=Error previous_ends_at=2024-01-10T04:05:30+05:30 next_ends_at=2024-01-10T04:07:30+05:30
logger=ngalert.notifier.alertmanager 1=(MISSING) t=2024-01-10T04:05:41.649843474+05:30 level=debug component=alertmanager orgID=1 msg="Putting alert" alert=DatasourceError[305dd1d][active] starts_at=2024-01-10T04:05:30+05:30 ends_at=2024-01-10T04:07:30+05:30
logger=ngalert.notifier.alertmanager 1=(MISSING) t=2024-01-10T04:05:41.815364371+05:30 level=debug component=alertmanager orgID=1 msg="Putting alert" alert=DatasourceError[d68547e][active] starts_at=2024-01-10T04:05:30+05:30 ends_at=2024-01-10T04:07:30+05:30
logger=ngalert.notifier.alertmanager 1=(MISSING) t=2024-01-10T04:05:42.043493465+05:30 level=debug component=alertmanager orgID=1 msg="Putting alert" alert=DatasourceError[d1a0459][active] starts_at=2024-01-10T04:05:30+05:30 ends_at=2024-01-10T04:07:30+05:30
logger=ngalert.notifier.alertmanager 1=(MISSING) t=2024-01-10T04:05:42.151338353+05:30 level=debug component=alertmanager orgID=1 msg="Putting alert" alert=DatasourceError[71ab0da][active] starts_at=2024-01-10T04:05:30+05:30 ends_at=2024-01-10T04:07:30+05:30
logger=ngalert.state.manager rule_uid=b3f0e71d-259f-4d3e-b0c4-cf15246e3a1a org_id=1 instance= t=2024-01-10T04:05:42.243824022+05:30 level=debug msg="Changing state" previous_state=Normal next_state=Error previous_ends_at=2024-01-10T04:05:30+05:30 next_ends_at=2024-01-10T04:07:30+05:30
logger=ngalert.notifier.alertmanager 1=(MISSING) t=2024-01-10T04:05:42.331357478+05:30 level=debug component=alertmanager orgID=1 msg="Putting alert" alert=DatasourceError[465515d][active] starts_at=2024-01-10T04:05:30+05:30 ends_at=2024-01-10T04:07:30+05:30
logger=ngalert.notifier.alertmanager 1=(MISSING) t=2024-01-10T04:05:42.614304517+05:30 level=debug component=alertmanager orgID=1 msg="Putting alert" alert=DatasourceError[bee9bd0][active] starts_at=2024-01-10T04:05:30+05:30 ends_at=2024-01-10T04:07:30+05:30
logger=ngalert.notifier.alertmanager 1=(MISSING) t=2024-01-10T04:05:42.994733279+05:30 level=debug component=alertmanager orgID=1 msg="Putting alert" alert=DatasourceError[ceadd91][active] starts_at=2024-01-10T04:05:30+05:30 ends_at=2024-01-10T04:07:30+05:30
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=f55ddd12-0516-41d1-8652-79030676b0b5 org_id=1 t=2024-01-10T04:06:30.27122652+05:30 level=debug msg="Sending query" start=2024-01-10T04:05:30+05:30 end=2024-01-10T04:06:30+05:30 step=15s query="probe_success{instance=~\"http://192\\\\.168\\\\.1\\\\.166:4042\"}"
log
What I would suggest is to delete the alert rule which is causing this error and also the one which you shared in the screenshot (as both of them seems a bit different) and then create a new one with same rules.
Sometimes this helps as old alert rules may still use old technique so try it out and do let us know.
Hi,
We got false mails again today at 10:46am
i got the logs in my grafana.log
logger=tsdb.prometheus endpoint=queryData pluginId=prometheus dsName=Prometheus dsUID=e78e23de-a513-4950-ad1e-ba026ec2824d uname=grafana_scheduler rule_uid=ffeeb4f0-9e70-4ab6-b248-fe784a865059 org_id=1 t=2024-01-24T10:46:11.638592517+05:30 level=debug msg="Sending query" start=2024-01-24T10:45:00+05:30 end=2024-01-24T10:46:00+05:30 step=15s query="(sum by(instance) (irate(node_cpu_seconds_total{instance=\"192.168.1.163:9100\",job=\"Unix_monitoring\", mode!=\"idle\"}[1m0s])) / on(instance) group_left sum by (instance)((irate(node_cpu_seconds_total{instance=\"192.168.1.163:9100\",job=\"Unix_monitoring\"}[1m0s])))) * 100"
logger=ngalert.scheduler rule_uid=ffeeb4f0-9e70-4ab6-b248-fe784a865059 org_id=1 version=13 fingerprint=5fab7d7bed572ba7 attempt=1 now=2024-01-24T10:46:00+05:30 rule_uid=ffeeb4f0-9e70-4ab6-b248-fe784a865059 org_id=1 t=2024-01-24T10:46:11.645202046+05:30 level=debug msg="Alert rule evaluated" results="[{Instance:instance=192.168.1.163:9100 State:Normal Error:<nil> Results:map[] Values:map[A:{Var:A Labels:instance=192.168.1.163:9100 Value:0xc002d31328} B:{Var:B Labels:instance=192.168.1.163:9100 Value:0xc002d31328} C:{Var:C Labels:instance=192.168.1.163:9100 Value:0xc002d31380}] EvaluatedAt:2024-01-24 10:46:00 +0530 IST EvaluationDuration:11.645187162s EvaluationString:[ var='A' labels={instance=192.168.1.163:9100} value=64.26615004331741 ], [ var='B' labels={instance=192.168.1.163:9100} value=64.26615004331741 ], [ var='C' labels={instance=192.168.1.163:9100} value=0 ]}]" duration=6.682949355s
Kindly refer it.
We would appreciate it if you can open a bug report about this in our official GitHub repository using this link.
Provide all the information (you can simply copy/paste your data + screenshot from here to there).
Lastly, paste the link to your GitHub issue here in this post so that all other community users can also track it. I will also add more additional information to it 