Grafana errors was working fine before

jbm7893 · December 31, 2022, 5:39pm

Version 9.2.3 (commit: ec9c2ed029, branch: HEAD)

I’m on xubuntu 18.04. Grafana has been working fine for years. Now I get this when starting from terminal. It sometimes show different line errors so I’ll post two instances. Any help would be appreciated. I’m not a pro at linux though.

 grafana-server.service - Grafana instance
   Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; disabled; ven
   Active: active (running) since Sat 2022-12-31 01:29:37 CST; 9h ago
     Docs: http://docs.grafana.org
 Main PID: 2136 (grafana-server)
    Tasks: 13 (limit: 4915)
   CGroup: /system.slice/grafana-server.service
           └─2136 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --p

Dec 31 10:29:38 johnnydell grafana-server[2136]: logger=cleanup t=2022-12-31T10:
Dec 31 10:39:38 johnnydell grafana-server[2136]: logger=cleanup t=2022-12-31T10:
Dec 31 10:49:38 johnnydell grafana-server[2136]: logger=cleanup t=2022-12-31T10:
Dec 31 10:59:38 johnnydell grafana-server[2136]: logger=cleanup t=2022-12-31T10:
Dec 31 11:09:38 johnnydell grafana-server[2136]: logger=cleanup t=2022-12-31T11:
Dec 31 11:19:38 johnnydell grafana-server[2136]: logger=cleanup t=2022-12-31T11:
Dec 31 11:24:44 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.se
Dec 31 11:24:44 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.se
Dec 31 11:24:44 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.se
Dec 31 11:24:44 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.se

Another instance when starting

johnny@johnnydell:~$ sudo systemctl daemon-reload | sudo systemctl start grafana-server | sudo systemctl status grafana-server
● grafana-server.service - Grafana instance
   Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; disabled; vendor preset: enabled)
   Active: active (running) since Sat 2022-12-31 01:29:37 CST; 10h ago
     Docs: http://docs.grafana.org
 Main PID: 2136 (grafana-server)
    Tasks: 13 (limit: 4915)
   CGroup: /system.slice/grafana-server.service
           └─2136 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/run/grafana/grafana-server.pid --packaging=deb cfg:default.paths.logs=/var/log/grafana cfg:default.paths.data=/var/lib/grafana cfg:default.paths.plugins=/var/lib/grafana/plugins cfg:d

Dec 31 11:19:38 johnnydell grafana-server[2136]: logger=cleanup t=2022-12-31T11:19:38.729117178-06:00 level=info msg="Completed cleanup jobs" duration=5.46247ms
Dec 31 11:24:44 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.service:29: Unknown lvalue 'ProtectClock' in section 'Service'
Dec 31 11:24:44 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.service:32: Unknown lvalue 'ProtectHostname' in section 'Service'
Dec 31 11:24:44 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.service:33: Unknown lvalue 'ProtectKernelLogs' in section 'Service'
Dec 31 11:24:44 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.service:36: Unknown lvalue 'ProtectProc' in section 'Service'
Dec 31 11:29:38 johnnydell grafana-server[2136]: logger=cleanup t=2022-12-31T11:29:38.729422531-06:00 level=info msg="Completed cleanup jobs" duration=4.891356ms
Dec 31 11:37:01 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.service:29: Unknown lvalue 'ProtectClock' in section 'Service'
Dec 31 11:37:01 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.service:32: Unknown lvalue 'ProtectHostname' in section 'Service'
Dec 31 11:37:01 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.service:33: Unknown lvalue 'ProtectKernelLogs' in section 'Service'
Dec 31 11:37:01 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.service:36: Unknown lvalue 'ProtectProc' in section 'Service'
lines 1-19/19 (END)

jbm7893 · January 2, 2023, 3:11am

It started working again after 3 days. Not sure why the cleanup or whatever took so long. Is there something i need to do or is there a way to do a manual cleanup? Here’s one line. It took longer than that of course. Not sure about those other errors either.

grafana-server[2248]: logger=cleanup t=2023-01-01T19:10:08.749348569-06:00 level=info msg=“Completed cleanup jobs” duration=5.14436ms

usman.ahmad · January 2, 2023, 12:13pm

Hi @jbm7893,

Welcome to the community support forums !!

When was the last time you used Grafana before this message appeared?

Also, Could you please provide us with the backend logs (the default location in Linux is /var/log/grafana/grafana.log) so that we can have more information about the error message?

Thanks

jbm7893 · January 2, 2023, 4:56pm

I’ve been running it continuously for years for my weather sensors. This happened after a reboot. I run the commands below to start it after I reboot. All the errors and cleanup messages are gone now after 3 days.

rtl_433 -M protocol -M level -C si -F "influx://localhost:8086/write?db=weather&p=gBh2mt44a=weather"

sudo systemctl daemon-reload | sudo systemctl start grafana-server | sudo systemctl status grafana-server

Here is the log:

logger=cleanup t=2023-01-02T00:00:08.7485773-06:00 level=info msg="Completed cleanup jobs" duration=4.405613ms
logger=cleanup t=2023-01-02T00:10:08.749854658-06:00 level=info msg="Completed cleanup jobs" duration=5.452743ms
logger=cleanup t=2023-01-02T00:20:08.747706248-06:00 level=info msg="Completed cleanup jobs" duration=3.729896ms
logger=cleanup t=2023-01-02T00:30:08.749628804-06:00 level=info msg="Completed cleanup jobs" duration=4.859986ms
logger=cleanup t=2023-01-02T00:40:08.750272588-06:00 level=info msg="Completed cleanup jobs" duration=5.912342ms
logger=cleanup t=2023-01-02T00:50:08.748855337-06:00 level=info msg="Completed cleanup jobs" duration=4.667401ms
logger=cleanup t=2023-01-02T01:00:08.749981419-06:00 level=info msg="Completed cleanup jobs" duration=5.06768ms
logger=cleanup t=2023-01-02T01:10:08.750072789-06:00 level=info msg="Completed cleanup jobs" duration=5.629729ms
logger=cleanup t=2023-01-02T01:20:08.749526593-06:00 level=info msg="Completed cleanup jobs" duration=5.442175ms
logger=cleanup t=2023-01-02T01:30:08.750533153-06:00 level=info msg="Completed cleanup jobs" duration=5.772868ms
logger=cleanup t=2023-01-02T01:40:08.75000816-06:00 level=info msg="Completed cleanup jobs" duration=5.074179ms
logger=cleanup t=2023-01-02T01:50:08.749512017-06:00 level=info msg="Completed cleanup jobs" duration=5.246411ms
logger=cleanup t=2023-01-02T02:00:08.749841096-06:00 level=info msg="Completed cleanup jobs" duration=5.236292ms
logger=cleanup t=2023-01-02T02:10:08.750801312-06:00 level=info msg="Completed cleanup jobs" duration=5.91768ms
logger=cleanup t=2023-01-02T02:20:08.749417539-06:00 level=info msg="Completed cleanup jobs" duration=5.358459ms
logger=cleanup t=2023-01-02T02:30:08.750083897-06:00 level=info msg="Completed cleanup jobs" duration=5.155602ms
logger=cleanup t=2023-01-02T02:40:08.749494976-06:00 level=info msg="Completed cleanup jobs" duration=5.1216ms
logger=cleanup t=2023-01-02T02:50:08.7494249-06:00 level=info msg="Completed cleanup jobs" duration=5.132625ms
logger=cleanup t=2023-01-02T03:00:08.750421737-06:00 level=info msg="Completed cleanup jobs" duration=6.326386ms
logger=cleanup t=2023-01-02T03:10:08.750420509-06:00 level=info msg="Completed cleanup jobs" duration=5.91162ms
logger=cleanup t=2023-01-02T03:20:08.749026409-06:00 level=info msg="Completed cleanup jobs" duration=4.684331ms
logger=cleanup t=2023-01-02T03:30:08.751133867-06:00 level=info msg="Completed cleanup jobs" duration=6.300162ms
logger=cleanup t=2023-01-02T03:40:08.749683247-06:00 level=info msg="Completed cleanup jobs" duration=5.467671ms
logger=cleanup t=2023-01-02T03:50:08.749861901-06:00 level=info msg="Completed cleanup jobs" duration=4.974744ms
logger=cleanup t=2023-01-02T04:00:08.749719094-06:00 level=info msg="Completed cleanup jobs" duration=5.432857ms
logger=cleanup t=2023-01-02T04:10:08.749932351-06:00 level=info msg="Completed cleanup jobs" duration=5.478162ms
logger=cleanup t=2023-01-02T04:20:08.74912153-06:00 level=info msg="Completed cleanup jobs" duration=5.0225ms
logger=cleanup t=2023-01-02T04:30:08.749579174-06:00 level=info msg="Completed cleanup jobs" duration=5.341907ms
logger=cleanup t=2023-01-02T04:40:08.751363962-06:00 level=info msg="Completed cleanup jobs" duration=6.839886ms
logger=cleanup t=2023-01-02T04:50:08.749303937-06:00 level=info msg="Completed cleanup jobs" duration=5.022748ms
logger=cleanup t=2023-01-02T05:00:08.750173852-06:00 level=info msg="Completed cleanup jobs" duration=5.262689ms
logger=cleanup t=2023-01-02T05:10:08.752755027-06:00 level=info msg="Completed cleanup jobs" duration=8.125279ms
logger=cleanup t=2023-01-02T05:20:08.750206772-06:00 level=info msg="Completed cleanup jobs" duration=5.416142ms
logger=cleanup t=2023-01-02T05:30:08.749079202-06:00 level=info msg="Completed cleanup jobs" duration=4.990853ms
logger=cleanup t=2023-01-02T05:40:08.749973159-06:00 level=info msg="Completed cleanup jobs" duration=5.056008ms
logger=cleanup t=2023-01-02T05:50:08.749234822-06:00 level=info msg="Completed cleanup jobs" duration=4.803147ms
logger=cleanup t=2023-01-02T06:00:08.74983947-06:00 level=info msg="Completed cleanup jobs" duration=5.333934ms
logger=cleanup t=2023-01-02T06:10:08.749841157-06:00 level=info msg="Completed cleanup jobs" duration=4.859475ms
logger=cleanup t=2023-01-02T06:20:08.749170849-06:00 level=info msg="Completed cleanup jobs" duration=4.844479ms
logger=cleanup t=2023-01-02T06:30:08.749050937-06:00 level=info msg="Completed cleanup jobs" duration=4.684444ms
logger=cleanup t=2023-01-02T06:40:08.749987213-06:00 level=info msg="Completed cleanup jobs" duration=5.422494ms
logger=cleanup t=2023-01-02T06:50:08.749554309-06:00 level=info msg="Completed cleanup jobs" duration=4.662581ms
logger=cleanup t=2023-01-02T07:00:08.749261951-06:00 level=info msg="Completed cleanup jobs" duration=5.401036ms
logger=cleanup t=2023-01-02T07:10:08.749986813-06:00 level=info msg="Completed cleanup jobs" duration=5.030322ms
logger=cleanup t=2023-01-02T07:20:08.749635398-06:00 level=info msg="Completed cleanup jobs" duration=5.771157ms
logger=cleanup t=2023-01-02T07:30:08.749084172-06:00 level=info msg="Completed cleanup jobs" duration=5.149483ms
logger=cleanup t=2023-01-02T07:40:08.751504107-06:00 level=info msg="Completed cleanup jobs" duration=7.124938ms
logger=cleanup t=2023-01-02T07:50:08.748856694-06:00 level=info msg="Completed cleanup jobs" duration=4.931723ms
logger=cleanup t=2023-01-02T08:00:08.749710257-06:00 level=info msg="Completed cleanup jobs" duration=5.030832ms
logger=cleanup t=2023-01-02T08:10:08.749791314-06:00 level=info msg="Completed cleanup jobs" duration=4.783929ms
logger=cleanup t=2023-01-02T08:20:08.749455486-06:00 level=info msg="Completed cleanup jobs" duration=5.000988ms
logger=cleanup t=2023-01-02T08:30:08.749955007-06:00 level=info msg="Completed cleanup jobs" duration=5.4002ms
logger=cleanup t=2023-01-02T08:40:08.749960736-06:00 level=info msg="Completed cleanup jobs" duration=5.203928ms
logger=cleanup t=2023-01-02T08:50:08.749893084-06:00 level=info msg="Completed cleanup jobs" duration=5.635198ms
logger=cleanup t=2023-01-02T09:00:08.749502858-06:00 level=info msg="Completed cleanup jobs" duration=5.352612ms
logger=cleanup t=2023-01-02T09:10:08.75026188-06:00 level=info msg="Completed cleanup jobs" duration=6.229246ms
logger=cleanup t=2023-01-02T09:20:08.7491425-06:00 level=info msg="Completed cleanup jobs" duration=5.284595ms
logger=cleanup t=2023-01-02T09:30:08.749766018-06:00 level=info msg="Completed cleanup jobs" duration=5.882758ms
logger=cleanup t=2023-01-02T09:40:08.749460235-06:00 level=info msg="Completed cleanup jobs" duration=5.191149ms
logger=cleanup t=2023-01-02T09:50:08.751058212-06:00 level=info msg="Completed cleanup jobs" duration=6.323773ms
logger=cleanup t=2023-01-02T10:00:08.750556346-06:00 level=info msg="Completed cleanup jobs" duration=5.773108ms
logger=cleanup t=2023-01-02T10:10:08.750618056-06:00 level=info msg="Completed cleanup jobs" duration=5.75686ms
logger=cleanup t=2023-01-02T10:20:08.750460886-06:00 level=info msg="Completed cleanup jobs" duration=6.228821ms
logger=cleanup t=2023-01-02T10:30:08.749646249-06:00 level=info msg="Completed cleanup jobs" duration=5.366772ms
logger=cleanup t=2023-01-02T10:40:08.749555219-06:00 level=info msg="Completed cleanup jobs" duration=5.028281ms
logger=cleanup t=2023-01-02T10:50:08.749421249-06:00 level=info msg="Completed cleanup jobs" duration=4.817058ms

jbm7893 · January 2, 2023, 5:25pm

It stopped working again.

 grafana-server.service - Grafana instance
   Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; disabled; vendor preset: enabled)
   Active: active (running) since Sat 2022-12-31 15:10:07 CST; 1 day 20h ago
     Docs: http://docs.grafana.org
 Main PID: 2248 (grafana-server)
    Tasks: 15 (limit: 4915)
   CGroup: /system.slice/grafana-server.service
           └─2248 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/run/grafana/grafana-server.pid --packaging=deb cfg:default.paths.logs=/var/log/grafana cfg:default.paths.data=/var/lib/grafana cfg:default.paths.plugins=/var/lib/grafana/plugins cfg:d

Jan 02 10:30:08 johnnydell grafana-server[2248]: logger=cleanup t=2023-01-02T10:30:08.749646249-06:00 level=info msg="Completed cleanup jobs" duration=5.366772ms
Jan 02 10:40:08 johnnydell grafana-server[2248]: logger=cleanup t=2023-01-02T10:40:08.749555219-06:00 level=info msg="Completed cleanup jobs" duration=5.028281ms
Jan 02 10:50:08 johnnydell grafana-server[2248]: logger=cleanup t=2023-01-02T10:50:08.749421249-06:00 level=info msg="Completed cleanup jobs" duration=4.817058ms
Jan 02 11:00:08 johnnydell grafana-server[2248]: logger=cleanup t=2023-01-02T11:00:08.749779769-06:00 level=info msg="Completed cleanup jobs" duration=5.603884ms
Jan 02 11:10:08 johnnydell grafana-server[2248]: logger=cleanup t=2023-01-02T11:10:08.751400969-06:00 level=info msg="Completed cleanup jobs" duration=6.472161ms
Jan 02 11:20:08 johnnydell grafana-server[2248]: logger=cleanup t=2023-01-02T11:20:08.749816293-06:00 level=info msg="Completed cleanup jobs" duration=4.998358ms
Jan 02 11:20:39 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.service:29: Unknown lvalue 'ProtectClock' in section 'Service'
Jan 02 11:20:39 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.service:32: Unknown lvalue 'ProtectHostname' in section 'Service'
Jan 02 11:20:39 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.service:33: Unknown lvalue 'ProtectKernelLogs' in section 'Service'
Jan 02 11:20:39 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.service:36: Unknown lvalue 'ProtectProc' in section 'Service'

jbm7893 · January 2, 2023, 5:28pm

Sorry for all the post but it wouldn’t let me post all the logs. Here they are.

https://pastebin.com/raw/ect8pHas

usman.ahmad · January 2, 2023, 6:01pm

Hi,

Thanks for the pastebin

I am not able to tell as why so many cleanup jobs are appearing and what triggers it.

Please change the default log level to debug mode. To do this go to the server where grafana is installed and edit the configuration file (/var/log/grafana/grafana.log)

There you will this section;

#################################### Logging ##########################
[log]
# Either "console", "file", "syslog". Default is console and  file
# Use space to separate multiple modes, e.g. "console file"
;mode = console file

# Either "debug", "info", "warn", "error", "critical", default is "info"
;level = info

Set the parameter level to debug and remove the colon in the beginning i.e.

level = debug

Then restart the grafana service and resent the logs once the issue occurs again.

jbm7893 · January 2, 2023, 6:18pm

Thanks for you help…I couldn’t find that in the log location. I edited it here /usr/local/etc/grafana/grafana.ini

The cleanup files are gone but now all the lines are this. These were showing in the first post I made but not all the lines.

● grafana-server.service - Grafana instance
Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: http://docs.grafana.org

Jan 02 12:12:33 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.service:29: Unknown lvalue ‘ProtectClock’ in section ‘Service’
Jan 02 12:12:33 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.service:32: Unknown lvalue ‘ProtectHostname’ in section ‘Service’
Jan 02 12:12:33 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.service:33: Unknown lvalue ‘ProtectKernelLogs’ in section ‘Service’
Jan 02 12:12:33 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.service:36: Unknown lvalue ‘ProtectProc’ in section ‘Service’
Jan 02 12:12:33 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.service:29: Unknown lvalue ‘ProtectClock’ in section ‘Service’
Jan 02 12:12:33 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.service:32: Unknown lvalue ‘ProtectHostname’ in section ‘Service’
Jan 02 12:12:33 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.service:33: Unknown lvalue ‘ProtectKernelLogs’ in section ‘Service’
Jan 02 12:12:33 johnnydell systemd[1]: /usr/lib/systemd/system/grafana-server.service:36: Unknown lvalue ‘ProtectProc’ in section ‘Service’
Jan 02 12:12:33 johnnydell systemd[1]: Started Grafana instance.
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
lines 1-14/14 (END)