Getting the following error in the logs:
logger=ngalert.scheduler rule_uid=e1a9e39f-08be-4ccb-b35a-c31279671446 org_id=1 version=12 fingerprint=ffdbbdb1a0206bcf attempt=0 now=2023-08-17T19:24:00Z t=2023-08-17T19:24:11.390450307Z level=error msg=“Failed to evaluate rule” error=“failed to execute query A: failed to convert long to wide series when converting from dataframe: long series must be sorted ascending by time to be converted” duration=2.366151292s
Don’t know what alert it is referring to. Also getting this soon after:
logger=sqlstore.transactions t=2023-08-17T19:23:27.33063325Z level=info msg=“Database locked, sleeping then retrying” error=“database is locked” retry=0 code=“database is locked”
Hello,
The first error tells you to sort the “time” color in ascending order in you query. This might fix the second error.
Can you show us your query?
1 Like
No because I don’t know what alert is doing it. There have been no new alerts added in days
There are also no errors showing in alerts
What is the source grafana server for this alert? Do you have another test server you use?
And where did you get that error log from
RPM version for Red Hat 8. Version 10.0.0
ogger=ngalert.state.manager rule_uid=ab522d30-9978-4980-8e9c-9ab5cc4a8050 org_id=1 t=2023-08-17T20:00:55.210253682Z level=error msg=“Error in expanding template” error=“failed to expand template ‘{{- $labels := .Labels -}}{{- $values := .Values -}}{{- $value := .Value -}}{{if gt $values.B.Value 90.0 }}critical{{else}}warning{{end}}’: error executing template __alert_Host Disk /home : template: __alert_Host Disk /home :1:81: executing "__alert_Host Disk /home " at <gt $values.B.Value 90.0>: error calling gt: invalid type for comparison”
logger=login_attempt t=2023-08-17T20:00:57.699201291Z level=error msg=“failed to lock and execute cleanup of old login attempts” error=“[sqlstore.max-retries-reached] retry 1: database is locked”
logger=sqlstore.transactions rule_uid=cc576892-f9f9-49e3-a98c-bc57a97341ab org_id=1 t=2023-08-17T20:00:57.405193683Z level=info msg=“Database locked, sleeping then retrying” error=“database is locked” retry=1 code=“database is locked”
logger=grafana.update.checker t=2023-08-17T20:00:58.079600776Z level=error msg=“Update check failed” error=“failed to get latest.json repo from github.com: Get "https://raw.githubusercontent.com/grafana/grafana/main/latest.json\”: dial tcp: lookup raw.githubusercontent.com: i/o timeout" duration=16.104710238s
logger=ngalert.state.manager rule_uid=af13af13-0621-411f-8d7e-ab3a9547e9fe org_id=1 t=2023-08-17T20:00:57.46062887Z level=error msg=“Failed to save alert state” labels=“alert_rule_namespace_uid=b0bd09e6-76e4-4b64-bf8f-fffdecdb8f86, alert_rule_uid=af13af13-0621-411f-8d7e-ab3a9547e9fe, alertname=Host Disk /usr, grafana_folder=Hosts, host=rh8” state=“unsupported value type” error=“[sqlstore.max-retries-reached] retry 1: database is locked”
logger=ngalert.state.manager rule_uid=e4a8e532-5fc3-498e-9912-d6cd0478bca3 org_id=1 t=2023-08-17T20:00:57.445336906Z level=error msg=“Failed to save alert state” labels=“alert_rule_namespace_uid=b0bd09e6-76e4-4b64-bf8f-fffdecdb8f86, alert_rule_uid=e4a8e532-5fc3-498e-9912-d6cd0478bca3, alertname=Rsyslog Status , grafana_folder=Hosts, host=rh8” state=“unsupported value type” error=“[sqlstore.max-retries-reached] retry 1: database is locked”
logger=sqlstore.transactions rule_uid=c8e0c5df-f830-442b-8d39-65c84cfd7885 org_id=1 t=2023-08-17T20:00:58.460241024Z level=info msg=“Database locked, sleeping then retrying” error=“database is locked” retry=1 code=“database is locked”
logger=ngalert.state.manager rule_uid=cc576892-f9f9-49e3-a98c-bc57a97341ab org_id=1 t=2023-08-17T20:00:58.769933795Z level=error msg=“Failed to save alert state” labels=“alert_rule_namespace_uid=b0bd09e6-76e4-4b64-bf8f-fffdecdb8f86, alert_rule_uid=cc576892-f9f9-49e3-a98c-bc57a97341ab, alertname=Crond Status , grafana_folder=Hosts, host=rh8” state=“unsupported value type” error=“[sqlstore.max-retries-reached] retry 1: database is locked”
logger=ngalert.state.manager rule_uid=c8e0c5df-f830-442b-8d39-65c84cfd7885 org_id=1 t=2023-08-17T20:02:46.380548807Z level=error msg=“Failed to save alert state” labels=“alert_rule_namespace_uid=b0bd09e6-76e4-4b64-bf8f-fffdecdb8f86, alert_rule_uid=c8e0c5df-f830-442b-8d39-65c84cfd7885, alertname=CPU Usage Critical, grafana_folder=Hosts, host=rh8crit” state=“unsupported value type” error=“[sqlstore.max-retries-reached] retry 1: database is locked”
logger=sqlstore.transactions t=2023-08-17T20:02:46.287172279Z level=info msg=“Database locked, sleeping then retrying” error=“database is locked” retry=0 code=“database is locked”
not sure if you missed this question above?
What is the source grafana server for this alert, meaning do you know which grafana server you have in your network this alert is coming from? Do you have another test server you use in your network or locally?
That’s it. I don’t know what alert is causing this
If you don’t know where it is coming from how can we? 
I see two possible hints that might nail it down
Rsyslog Status
Cron Status
Copy your grafana.db and query it to see which alerts those are associated with
Okay. I also checked the alert JSON and can’t find that alert ID anywhere
where is it from, who created it? Also that UUID you are filtering on does not seem to be associated with anything in your error logs.
I migrated to MySQL and the locking issues have gone away
2 Likes
