Grafana instance fails to start

wnasich · October 10, 2021, 11:57pm

Hi @niceguysan ,

This comment on github helped me to solve this issue:

github.com/grafana/grafana

After upgrade to version 8.2.0 can't start grafana service

opened 07:57AM - 08 Oct 21 UTC

closed 10:37AM - 08 Oct 21 UTC

IgorSimic

After upgrade to version 8.2.0 can't start grafana service. Installation via ap…t-get update was fine and at the end, service was restarted but never go up again I found that service file /usr/lib/systemd/system/grafana-server.service was changed and whole bunch of protection settings was added. Some of this new settings instruct kernel to kill grafana-server proccess. When I use old systemd service file, grafana run just fine. Below are informations about my system, log and "bad" & "good" systemd service file. $ uname -a Linux zabbix 5.10.71-sunxi64 #21.11.3 SMP Thu Oct 7 23:52:46 CEST 2021 aarch64 aarch64 aarch64 GNU/Linux $ cat /etc/issue Ubuntu 18.04.6 LTS \n \l Welcome to Ubuntu Bionic with Armbian Linux 5.10.71-sunxi64 $ dpkg -l | grep grafana ii grafana 8.2.0 arm64 Grafana syslog: Oct 8 08:20:33 localhost systemd[1]: Started Grafana instance. Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Main process exited, code=killed, status=31/SYS Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Failed with result 'signal'. Oct 8 08:20:33 localhost kernel: [ 7618.317684] audit: type=1326 audit(1633674033.139:18): auid=4294967295 uid=116 gid=124 ses=4294967295 subj==unconfined pid=21346 comm="grafana-server" exe="/usr/sbin/grafana-server" sig=31 arch=c00000b7 syscall=214 compat=0 ip=0xffffb465ecf4 code=0x0 Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Service hold-off time over, scheduling restart. Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 1. Oct 8 08:20:33 localhost systemd[1]: Stopped Grafana instance. Oct 8 08:20:33 localhost systemd[1]: Started Grafana instance. Oct 8 08:20:33 localhost kernel: [ 7618.587286] audit: type=1326 audit(1633674033.407:19): auid=4294967295 uid=116 gid=124 ses=4294967295 subj==unconfined pid=21351 comm="grafana-server" exe="/usr/sbin/grafana-server" sig=31 arch=c00000b7 syscall=214 compat=0 ip=0xffff80f02cf4 code=0x0 Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Main process exited, code=killed, status=31/SYS Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Failed with result 'signal'. Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Service hold-off time over, scheduling restart. Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 2. Oct 8 08:20:33 localhost systemd[1]: Stopped Grafana instance. Oct 8 08:20:33 localhost systemd[1]: Started Grafana instance. Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Main process exited, code=killed, status=31/SYS Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Failed with result 'signal'. Oct 8 08:20:33 localhost kernel: [ 7618.836798] audit: type=1326 audit(1633674033.659:20): auid=4294967295 uid=116 gid=124 ses=4294967295 subj==unconfined pid=21356 comm="grafana-server" exe="/usr/sbin/grafana-server" sig=31 arch=c00000b7 syscall=214 compat=0 ip=0xffff95418cf4 code=0x0 Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Service hold-off time over, scheduling restart. Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 3. Oct 8 08:20:33 localhost systemd[1]: Stopped Grafana instance. Oct 8 08:20:33 localhost systemd[1]: Started Grafana instance. Oct 8 08:20:33 localhost kernel: [ 7619.087799] audit: type=1326 audit(1633674033.907:21): auid=4294967295 uid=116 gid=124 ses=4294967295 subj==unconfined pid=21360 comm="grafana-server" exe="/usr/sbin/grafana-server" sig=31 arch=c00000b7 syscall=214 compat=0 ip=0xffffbcb75cf4 code=0x0 Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Main process exited, code=killed, status=31/SYS Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Failed with result 'signal'. Oct 8 08:20:34 localhost systemd[1]: grafana-server.service: Service hold-off time over, scheduling restart. Oct 8 08:20:34 localhost systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 4. Oct 8 08:20:34 localhost systemd[1]: Stopped Grafana instance. Oct 8 08:20:34 localhost systemd[1]: Started Grafana instance. Oct 8 08:20:34 localhost systemd[1]: grafana-server.service: Main process exited, code=killed, status=31/SYS Oct 8 08:20:34 localhost systemd[1]: grafana-server.service: Failed with result 'signal'. Oct 8 08:20:34 localhost kernel: [ 7619.341092] audit: type=1326 audit(1633674034.163:22): auid=4294967295 uid=116 gid=124 ses=4294967295 subj==unconfined pid=21364 comm="grafana-server" exe="/usr/sbin/grafana-server" sig=31 arch=c00000b7 syscall=214 compat=0 ip=0xffff86bbccf4 code=0x0 Oct 8 08:20:34 localhost systemd[1]: grafana-server.service: Service hold-off time over, scheduling restart. Oct 8 08:20:34 localhost systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 5. Oct 8 08:20:34 localhost systemd[1]: Stopped Grafana instance. Oct 8 08:20:34 localhost systemd[1]: grafana-server.service: Start request repeated too quickly. Oct 8 08:20:34 localhost systemd[1]: grafana-server.service: Failed with result 'signal'. Oct 8 08:20:34 localhost systemd[1]: Failed to start Grafana instance. Service file /usr/lib/systemd/system/grafana-server.service "Bad systemd service file": [Unit] Description=Grafana instance Documentation=http://docs.grafana.org Wants=network-online.target After=network-online.target After=postgresql.service mariadb.service mysql.service [Service] EnvironmentFile=/etc/default/grafana-server User=grafana Group=grafana Type=simple Restart=on-failure WorkingDirectory=/usr/share/grafana RuntimeDirectory=grafana RuntimeDirectoryMode=0750 ExecStart=/usr/sbin/grafana-server \ --config=${CONF_FILE} \ --pidfile=${PID_FILE_DIR}/grafana-server.pid \ --packaging=deb \ cfg:default.paths.logs=${LOG_DIR} \ cfg:default.paths.data=${DATA_DIR} \ cfg:default.paths.plugins=${PLUGINS_DIR} \ cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR} LimitNOFILE=10000 TimeoutStopSec=20 CapabilityBoundingSet= DeviceAllow= LockPersonality=true MemoryDenyWriteExecute=true NoNewPrivileges=true PrivateDevices=true PrivateTmp=true PrivateUsers=true ProcSubset=pid ProtectClock=true ProtectControlGroups=true ProtectHome=true ProtectHostname=true ProtectKernelLogs=true ProtectKernelModules=true ProtectKernelTunables=true ProtectProc=invisible ProtectSystem=full RemoveIPC=true RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX RestrictNamespaces=true RestrictRealtime=true RestrictSUIDSGID=true SystemCallArchitectures=native SystemCallFilter=@system-service SystemCallFilter=~@privileged SystemCallFilter=~@resources UMask=0027 [Install] WantedBy=multi-user.target "Good", old systemd service file": [Unit] Description=Grafana instance Documentation=http://docs.grafana.org Wants=network-online.target After=network-online.target After=postgresql.service mariadb.service mysql.service [Service] EnvironmentFile=/etc/default/grafana-server User=grafana Group=grafana Type=simple Restart=on-failure WorkingDirectory=/usr/share/grafana RuntimeDirectory=grafana RuntimeDirectoryMode=0750 ExecStart=/usr/sbin/grafana-server \ --config=${CONF_FILE} \ --pidfile=${PID_FILE_DIR}/grafana-server.pid \ --packaging=deb \ cfg:default.paths.logs=${LOG_DIR} \ cfg:default.paths.data=${DATA_DIR} \ cfg:default.paths.plugins=${PLUGINS_DIR} \ cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR} LimitNOFILE=10000 TimeoutStopSec=20 UMask=0027 [Install] WantedBy=multi-user.target

I am running Grafana on Ubuntu 16.04.7 LTS , it seems related to the use of a systemd function not available in old versions

Topic		Replies	Views
Grafana stopped working after an update to 8.2.1 Installation	2	5987	November 11, 2021
Grafana server not start (after 8.2.0 upgrade) Installation	14	12037	April 19, 2022
Grafana service failed to start though running before Installation	1	8396	August 26, 2021
Grafana Service start error on upgrade to v8.4.3 Installation	0	1070	March 10, 2022
Grafana Service Not Starting Configuration	18	14304	January 11, 2023

Grafana instance fails to start

Related topics