Grafana fails to start with mkdir file exists error

Grafana Configuration
1

  • What Grafana version and what operating system are you using?
    9.2.1.0

  • What are you trying to achieve?
    Grafana was working fine, now after reboot will not start from systemd but starts and runs ok manually via cli

Grafana was working fine on my RedHat system, and had even been restarted via systemctl not too long ago. The system was rebooted, possibly due to patching (SOE is under management), now Grafana will not start from systemctl, giving the below error:

Jun 21 11:19:37 hostname grafana-server[755749]: Failed to start grafana. error: failed to connect to database: mkdir /app: file exists
Jun 21 11:19:37 hostname grafana-server[755749]: failed to connect to database: mkdir /app: file exists
Jun 21 11:19:37 hostname systemd[1]: grafana-server.service: Main process exited, code=exited, status=1/FAILURE
Jun 21 11:19:37 hostname systemd[1]: grafana-server.service: Failed with result ‘exit-code’.

when I su to the grafana user, I can run the full systemd executable path and it runs fine:

/usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/app/grafana/grafana-server.pid --packaging=rpm cfg:default.paths.logs=/var/log/grafana cfg:default.paths.data=/var/lib/grafana cfg:default.paths.plugins=/var/lib/grafana/plugins cfg:default.paths.provisioning=/etc/grafana/provisioning

Here is the full systemd file

[Unit]
Description=Grafana instance
Documentation=http://docs.grafana.org
Wants=network-online.target
After=network-online.target
After=postgresql.service mariadb.service mysqld.service

[Service]
EnvironmentFile=/etc/sysconfig/grafana-server
User=grafana
Group=grafana
Type=notify
Restart=on-failure
WorkingDirectory=/usr/share/grafana
RuntimeDirectory=grafana
RuntimeDirectoryMode=0750
ExecStart=/usr/sbin/grafana-server
–config=${CONF_FILE}
–pidfile=${PID_FILE_DIR}/grafana-server.pid
–packaging=rpm
cfg:default.paths.logs=${LOG_DIR}
cfg:default.paths.data=${DATA_DIR}
cfg:default.paths.plugins=${PLUGINS_DIR}
cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR}

LimitNOFILE=10000
TimeoutStopSec=20
CapabilityBoundingSet=
DeviceAllow=
LockPersonality=true
MemoryDenyWriteExecute=false
NoNewPrivileges=true
PrivateDevices=true
PrivateTmp=true
ProtectClock=true
ProtectControlGroups=true
ProtectHome=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectProc=invisible
ProtectSystem=full
RemoveIPC=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true
SystemCallArchitectures=native
UMask=0027

[Install]
WantedBy=multi-user.target

and the env file

GRAFANA_USER=grafana

GRAFANA_GROUP=grafana

GRAFANA_HOME=/usr/share/grafana

LOG_DIR=/var/log/grafana

DATA_DIR=/var/lib/grafana

MAX_OPEN_FILES=10000

CONF_DIR=/etc/grafana

CONF_FILE=/etc/grafana/grafana.ini

RESTART_ON_UPGRADE=true

PLUGINS_DIR=/var/lib/grafana/plugins

PROVISIONING_CFG_DIR=/etc/grafana/provisioning

# Only used on systemd systems
#PID_FILE_DIR=/var/run/grafana
PID_FILE_DIR=/app/grafana

I cannot find anything more helpful in the logs, why is this failing via systemd? Sorry I’m a bit stuck and need some help!

thanks.