Grafana service does not start after reboot

crichmond · September 14, 2017, 7:51pm

Have been recently working to get a new instance of Icinga2 working, and added InfluxDB/grafana modules.

All (including grafana) has been working since mid-last week (when I got it all working correctly).

Ubuntu 16.04 LTS
This morning, did a reboot of the server (first full reboot since getting it all working), and now grafana does not start on boot.
First noticed when the pages within IcingaWeb said it could connect to the server:port for the graphs.
Tried manually connecting to grafana web interface at servername:port, but no luck
Other services on machine appear OK.

Output of $ sudo service grafana status and $ sudo systemctl status grafana.service result in same:
● grafana.service - LSB: Start grafana at boot time
Loaded: loaded (/etc/init.d/grafana; bad; vendor preset: enabled)
Active: failed (Result: resources) since Thu 2017-09-14 14:04:15 CDT; 36min ago
Docs: man:systemd-sysv-generator(8)
Process: 13546 ExecStart=/etc/init.d/grafana start (code=exited, status=0/SUCCESS)

Sep 14 14:04:15 systemd[1]: Starting LSB: Start grafana at boot time…
Sep 14 14:04:15 systemd[1]: grafana.service: PID file /var/run/grafana.pid not readable (yet?) after start: No suc
Sep 14 14:04:15 systemd[1]: Failed to start LSB: Start grafana at boot time.
Sep 14 14:04:15 systemd[1]: grafana.service: Unit entered failed state.
Sep 14 14:04:15 systemd[1]: grafana.service: Failed with result ‘resources’.

I’m not a *nix expert by any means, but I can follow directions…most of the topics searching online so far about the “not readable (yet?)” message are several years old and appear to be related to when systemd as first coming out and had timing issues about creating .pid file for backwards compatibility.

Suggestions on where I go from here to troubleshoot?

crichmond · September 15, 2017, 9:13pm

There was an update available this morning from 4.5 to 4.5.1 After applying the update, grafana is now available again

crichmond · October 13, 2017, 3:13pm

Well, a restart this morning after updates, and now am getting the same issue again where grafana does not start.

Not accessible via web portal
Same output as above when running “service grafana status”

Any suggestions on a fix for getting grafana to start correctly after a reboot?

crichmond · October 17, 2017, 2:17pm

Appears to be the “grafana-server” service and not the “grafana” service. When showing a service status, it shows enabled, but not running. If I manually start the grafana-server service, then the site is back up and functioning again.

sjp770 · March 25, 2018, 9:55pm

Did you find a way to return it to autostarting though? I just had the same issue, but have had to manually start the service each reboot.

daniellee · March 27, 2018, 9:22am

Any errors in the Grafana server logs?

nightstorm · April 28, 2019, 4:39pm

I have also the same issue. Right now I’m not able to set “grafana-server” to start at boot-time.

Hardware, OS: RPi3, Raspian

Try1:
sudo update-rc.d grafana-server defaults

No error, but grafana is not starting after a reboot.

Try2: --> I do not realy understand the error

sudo systemctl enable grafana-server.service

Synchronizing state of grafana-server.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable grafana-server
Failed to enable unit: Unit /run/systemd/generator.late/grafana.service is transient or generated.

clanlaw · April 29, 2019, 7:41am

How did you install grafana?

What does this show

sudo systemctl status grafana-server

Copy/paste the complete output please.

nightstorm · April 29, 2019, 7:17pm

Installation:

wget https://dl.grafana.com/oss/release/grafana_6.1.6_armhf.deb
sudo dpkg -i grafana_6.1.6_armhf.deb

(https grafana com/grafana/download?platform=arm)

Systemctl: --> if startet grafana manuel. After reboot, grafana does not start automaticly

sudo systemctl status grafana-server

● grafana-server.service - Grafana instance
Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; generated; vendor pr
Active: active (running) since Sun 2019-04-28 16:41:49 UTC; 1 day 2h ago
Docs: http docs grafana org
Main PID: 747 (grafana-server)
CGroup: /system.slice/grafana-server.service
└─747 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/

Apr 29 19:02:24 raspberrypi grafana-server[747]: t=2019-04-29T19:02:24+0000 lvl=eror ms
Apr 29 19:02:24 raspberrypi grafana-server[747]: t=2019-04-29T19:02:24+0000 lvl=eror ms
Apr 29 19:02:26 raspberrypi grafana-server[747]: t=2019-04-29T19:02:26+0000 lvl=eror ms
Apr 29 19:02:26 raspberrypi grafana-server[747]: t=2019-04-29T19:02:26+0000 lvl=eror ms
Apr 29 19:02:26 raspberrypi grafana-server[747]: t=2019-04-29T19:02:26+0000 lvl=eror ms
Apr 29 19:02:26 raspberrypi grafana-server[747]: t=2019-04-29T19:02:26+0000 lvl=eror ms
Apr 29 19:02:32 raspberrypi grafana-server[747]: t=2019-04-29T19:02:32+0000 lvl=eror ms
Apr 29 19:02:32 raspberrypi grafana-server[747]: t=2019-04-29T19:02:32+0000 lvl=eror ms
Apr 29 19:02:32 raspberrypi grafana-server[747]: t=2019-04-29T19:02:32+0000 lvl=eror ms
Apr 29 19:02:32 raspberrypi grafana-server[747]: t=2019-04-29T19:02:32+0000 lvl=eror ms

Thanks for your support.

nightstorm · April 29, 2019, 7:39pm

I think we got a solution.

Step by Step

rename / delete grafana-server.service from /etc/systemd/system
sudo systemctl enable grafana-server.service
restart and test → be happy

Console
pi@raspberrypi:/etc/init.d $ cd /etc/systemd/system/
pi@raspberrypi:/etc/systemd/system $ ls
autologin@.service multi-user.target.wants
bluetooth.target.wants network-online.target.wants
dbus-org.bluez.service poweroff.target.wants
dbus-org.freedesktop.Avahi.service rc-local.service.d
dhcpcd5.service reboot.target.wants
dhcpcd.service.d remote-fs.target.wants
getty.target.wants sockets.target.wants
getty@tty1.service.d sysinit.target.wants
grafana-server.service syslog.service
halt.target.wants timers.target.wants
influxd.service
pi@raspberrypi:/etc/systemd/system $ mv grafana-server.service grafana-server.service_old
mv: cannot move ‘grafana-server.service’ to ‘grafana-server.service_old’: Permission denied
pi@raspberrypi:/etc/systemd/system $ sudo mv grafana-server.service grafana-server.service_old
pi@raspberrypi:/etc/systemd/system $ ls
autologin@.service multi-user.target.wants
bluetooth.target.wants network-online.target.wants
dbus-org.bluez.service poweroff.target.wants
dbus-org.freedesktop.Avahi.service rc-local.service.d
dhcpcd5.service reboot.target.wants
dhcpcd.service.d remote-fs.target.wants
getty.target.wants sockets.target.wants
getty@tty1.service.d sysinit.target.wants
grafana-server.service_old syslog.service
halt.target.wants timers.target.wants
influxd.service