Had Grafana running smoothly in rhel 8 for a while, then suddenly on morning it failed and doesn’t start.
systemctl start grafana-server
Job for grafana-server.service failed because the control process exited with error code.
See “systemctl status grafana-server.service” and “journalctl -xe” for details.
systemctl status grafana-server
● grafana-server.service - Grafana instance
Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; disabled; vendor preset: disabl>
Active: failed (Result: exit-code) since Thu 2021-08-26 00:04:50 EDT; 1h 44min ago
Docs: http://docs.grafana.org
Process: 2013 ExecStart=/usr/sbin/grafana-server --config=${CONF_FILE} --pidfile=${PID_FILE_DIR}>
Main PID: 2013 (code=exited, status=1/FAILURE)
Aug 26 00:04:49 t10.darway.lan systemd[1]: Failed to start Grafana instance.
Aug 26 00:04:50 t10.darway.lan systemd[1]: grafana-server.service: Service RestartSec=100ms expire>
Aug 26 00:04:50 t10.darway.lan systemd[1]: grafana-server.service: Scheduled restart job, restart >
Aug 26 00:04:50 t10.darway.lan systemd[1]: Stopped Grafana instance.
Aug 26 00:04:50 t10.darway.lan systemd[1]: grafana-server.service: Start request repeated too quic>
Aug 26 00:04:50 t10.darway.lan systemd[1]: grafana-server.service: Failed with result ‘exit-code’.
Aug 26 00:04:50 t10.darway.lan systemd[1]: Failed to start Grafana instance.
Aug 26 00:04:57 t10.darway.lan systemd[1]: grafana-server.service: Start request repeated too quic>
Aug 26 00:04:57 t10.darway.lan systemd[1]: grafana-server.service: Failed with result ‘exit-code’.
Aug 26 00:04:57 t10.darway.lan systemd[1]: Failed to start Grafana instance.
Note: /usr/lib/systemd/system/grafana-server.service is the original, no alterations.
#tail -f /var/log/grafana/grafana.log
t=2021-08-26T01:54:01-0400 lvl=info msg=“Starting Grafana” logger=server version=8.1.1 commit=90c87a52f7 branch=HEAD compiled=2021-08-09T08:53:33-0400
t=2021-08-26T01:54:01-0400 lvl=info msg=“Config loaded from” logger=settings file=/usr/share/grafana/conf/defaults.ini
t=2021-08-26T01:54:01-0400 lvl=info msg=“Config loaded from” logger=settings file=/etc/grafana/grafana.ini
t=2021-08-26T01:54:01-0400 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.data=/var/lib/grafana”
t=2021-08-26T01:54:01-0400 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.logs=/var/log/grafana”
t=2021-08-26T01:54:01-0400 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.plugins=/var/lib/grafana/plugins”
t=2021-08-26T01:54:01-0400 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.provisioning=/etc/grafana/provisioning”
t=2021-08-26T01:54:01-0400 lvl=info msg=“Path Home” logger=settings path=/usr/share/grafana
t=2021-08-26T01:54:01-0400 lvl=info msg=“Path Data” logger=settings path=/var/lib/grafana
t=2021-08-26T01:54:01-0400 lvl=info msg=“Path Logs” logger=settings path=/var/log/grafana
t=2021-08-26T01:54:01-0400 lvl=info msg=“Path Plugins” logger=settings path=/var/lib/grafana/plugins
t=2021-08-26T01:54:01-0400 lvl=info msg=“Path Provisioning” logger=settings path=/etc/grafana/provisioning
t=2021-08-26T01:54:01-0400 lvl=info msg=“App mode production” logger=settings
t=2021-08-26T01:54:01-0400 lvl=info msg=“Writing PID file” logger=server path=/var/run/grafana/grafana-server.pid pid=7332
t=2021-08-26T01:54:01-0400 lvl=info msg=“Connecting to DB” logger=sqlstore dbtype=sqlite3
t=2021-08-26T01:54:01-0400 lvl=info msg=“Starting DB migrations” logger=migrator
t=2021-08-26T01:54:01-0400 lvl=info msg=“migrations completed” logger=migrator performed=0 skipped=330 duration=503.218µs
t=2021-08-26T01:54:01-0400 lvl=info msg=“Starting plugin search” logger=plugins
t=2021-08-26T01:54:01-0400 lvl=info msg=“Registering plugin” logger=plugins id=input
t=2021-08-26T01:54:01-0400 lvl=info msg=“Starting Grafana” logger=server version=8.1.1 commit=90c87a52f7 branch=HEAD compiled=2021-08-09T08:53:33-0400
t=2021-08-26T01:54:01-0400 lvl=info msg=“Config loaded from” logger=settings file=/usr/share/grafana/conf/defaults.ini
t=2021-08-26T01:54:01-0400 lvl=info msg=“Config loaded from” logger=settings file=/etc/grafana/grafana.ini
t=2021-08-26T01:54:01-0400 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.data=/var/lib/grafana”
t=2021-08-26T01:54:01-0400 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.logs=/var/log/grafana”
t=2021-08-26T01:54:01-0400 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.plugins=/var/lib/grafana/plugins”
t=2021-08-26T01:54:01-0400 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.provisioning=/etc/grafana/provisioning”
t=2021-08-26T01:54:01-0400 lvl=info msg=“Path Home” logger=settings path=/usr/share/grafana
t=2021-08-26T01:54:01-0400 lvl=info msg=“Path Data” logger=settings path=/var/lib/grafana
t=2021-08-26T01:54:01-0400 lvl=info msg=“Path Logs” logger=settings path=/var/log/grafana
t=2021-08-26T01:54:01-0400 lvl=info msg=“Path Plugins” logger=settings path=/var/lib/grafana/plugins
t=2021-08-26T01:54:01-0400 lvl=info msg=“Path Provisioning” logger=settings path=/etc/grafana/provisioning
t=2021-08-26T01:54:01-0400 lvl=info msg=“App mode production” logger=settings
t=2021-08-26T01:54:01-0400 lvl=info msg=“Writing PID file” logger=server path=/var/run/grafana/grafana-server.pid pid=7341
t=2021-08-26T01:54:01-0400 lvl=info msg=“Connecting to DB” logger=sqlstore dbtype=sqlite3
t=2021-08-26T01:54:01-0400 lvl=info msg=“Starting DB migrations” logger=migrator
t=2021-08-26T01:54:01-0400 lvl=info msg=“migrations completed” logger=migrator performed=0 skipped=330 duration=536.475µs
t=2021-08-26T01:54:01-0400 lvl=info msg=“Starting plugin search” logger=plugins
t=2021-08-26T01:54:01-0400 lvl=info msg=“Registering plugin” logger=plugins id=input
t=2021-08-26T01:54:01-0400 lvl=info msg=“Starting Grafana” logger=server version=8.1.1 commit=90c87a52f7 branch=HEAD compiled=2021-08-09T08:53:33-0400
t=2021-08-26T01:54:01-0400 lvl=info msg=“Config loaded from” logger=settings file=/usr/share/grafana/conf/defaults.ini
t=2021-08-26T01:54:01-0400 lvl=info msg=“Config loaded from” logger=settings file=/etc/grafana/grafana.ini
t=2021-08-26T01:54:01-0400 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.data=/var/lib/grafana”
t=2021-08-26T01:54:01-0400 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.logs=/var/log/grafana”
t=2021-08-26T01:54:01-0400 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.plugins=/var/lib/grafana/plugins”
t=2021-08-26T01:54:01-0400 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.provisioning=/etc/grafana/provisioning”
t=2021-08-26T01:54:01-0400 lvl=info msg=“Path Home” logger=settings path=/usr/share/grafana
t=2021-08-26T01:54:01-0400 lvl=info msg=“Path Data” logger=settings path=/var/lib/grafana
t=2021-08-26T01:54:01-0400 lvl=info msg=“Path Logs” logger=settings path=/var/log/grafana
t=2021-08-26T01:54:01-0400 lvl=info msg=“Path Plugins” logger=settings path=/var/lib/grafana/plugins
t=2021-08-26T01:54:01-0400 lvl=info msg=“Path Provisioning” logger=settings path=/etc/grafana/provisioning
t=2021-08-26T01:54:01-0400 lvl=info msg=“App mode production” logger=settings
t=2021-08-26T01:54:01-0400 lvl=info msg=“Writing PID file” logger=server path=/var/run/grafana/grafana-server.pid pid=7351
t=2021-08-26T01:54:01-0400 lvl=info msg=“Connecting to DB” logger=sqlstore dbtype=sqlite3
t=2021-08-26T01:54:01-0400 lvl=info msg=“Starting DB migrations” logger=migrator
t=2021-08-26T01:54:01-0400 lvl=info msg=“migrations completed” logger=migrator performed=0 skipped=330 duration=801.724µs
t=2021-08-26T01:54:01-0400 lvl=info msg=“Starting plugin search” logger=plugins
t=2021-08-26T01:54:01-0400 lvl=info msg=“Registering plugin” logger=plugins id=input
t=2021-08-26T01:54:02-0400 lvl=info msg=“Starting Grafana” logger=server version=8.1.1 commit=90c87a52f7 branch=HEAD compiled=2021-08-09T08:53:33-0400
t=2021-08-26T01:54:02-0400 lvl=info msg=“Config loaded from” logger=settings file=/usr/share/grafana/conf/defaults.ini
t=2021-08-26T01:54:02-0400 lvl=info msg=“Config loaded from” logger=settings file=/etc/grafana/grafana.ini
t=2021-08-26T01:54:02-0400 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.data=/var/lib/grafana”
t=2021-08-26T01:54:02-0400 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.logs=/var/log/grafana”
t=2021-08-26T01:54:02-0400 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.plugins=/var/lib/grafana/plugins”
t=2021-08-26T01:54:02-0400 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.provisioning=/etc/grafana/provisioning”
t=2021-08-26T01:54:02-0400 lvl=info msg=“Path Home” logger=settings path=/usr/share/grafana
t=2021-08-26T01:54:02-0400 lvl=info msg=“Path Data” logger=settings path=/var/lib/grafana
t=2021-08-26T01:54:02-0400 lvl=info msg=“Path Logs” logger=settings path=/var/log/grafana
t=2021-08-26T01:54:02-0400 lvl=info msg=“Path Plugins” logger=settings path=/var/lib/grafana/plugins
t=2021-08-26T01:54:02-0400 lvl=info msg=“Path Provisioning” logger=settings path=/etc/grafana/provisioning
t=2021-08-26T01:54:02-0400 lvl=info msg=“App mode production” logger=settings
t=2021-08-26T01:54:02-0400 lvl=info msg=“Writing PID file” logger=server path=/var/run/grafana/grafana-server.pid pid=7360
t=2021-08-26T01:54:02-0400 lvl=info msg=“Connecting to DB” logger=sqlstore dbtype=sqlite3
t=2021-08-26T01:54:02-0400 lvl=info msg=“Starting DB migrations” logger=migrator
t=2021-08-26T01:54:02-0400 lvl=info msg=“migrations completed” logger=migrator performed=0 skipped=330 duration=517.307µs
t=2021-08-26T01:54:02-0400 lvl=info msg=“Starting plugin search” logger=plugins
t=2021-08-26T01:54:02-0400 lvl=info msg=“Registering plugin” logger=plugins id=input
t=2021-08-26T01:54:02-0400 lvl=info msg=“Starting Grafana” logger=server version=8.1.1 commit=90c87a52f7 branch=HEAD compiled=2021-08-09T08:53:33-0400
t=2021-08-26T01:54:02-0400 lvl=info msg=“Config loaded from” logger=settings file=/usr/share/grafana/conf/defaults.ini
t=2021-08-26T01:54:02-0400 lvl=info msg=“Config loaded from” logger=settings file=/etc/grafana/grafana.ini
t=2021-08-26T01:54:02-0400 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.data=/var/lib/grafana”
t=2021-08-26T01:54:02-0400 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.logs=/var/log/grafana”
t=2021-08-26T01:54:02-0400 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.plugins=/var/lib/grafana/plugins”
t=2021-08-26T01:54:02-0400 lvl=info msg=“Config overridden from command line” logger=settings arg=“default.paths.provisioning=/etc/grafana/provisioning”
t=2021-08-26T01:54:02-0400 lvl=info msg=“Path Home” logger=settings path=/usr/share/grafana
t=2021-08-26T01:54:02-0400 lvl=info msg=“Path Data” logger=settings path=/var/lib/grafana
t=2021-08-26T01:54:02-0400 lvl=info msg=“Path Logs” logger=settings path=/var/log/grafana
t=2021-08-26T01:54:02-0400 lvl=info msg=“Path Plugins” logger=settings path=/var/lib/grafana/plugins
t=2021-08-26T01:54:02-0400 lvl=info msg=“Path Provisioning” logger=settings path=/etc/grafana/provisioning
t=2021-08-26T01:54:02-0400 lvl=info msg=“App mode production” logger=settings
t=2021-08-26T01:54:02-0400 lvl=info msg=“Writing PID file” logger=server path=/var/run/grafana/grafana-server.pid pid=7369
t=2021-08-26T01:54:02-0400 lvl=info msg=“Connecting to DB” logger=sqlstore dbtype=sqlite3
t=2021-08-26T01:54:02-0400 lvl=info msg=“Starting DB migrations” logger=migrator
t=2021-08-26T01:54:02-0400 lvl=info msg=“migrations completed” logger=migrator performed=0 skipped=330 duration=461.486µs
t=2021-08-26T01:54:02-0400 lvl=info msg=“Starting plugin search” logger=plugins
t=2021-08-26T01:54:02-0400 lvl=info msg=“Registering plugin” logger=plugins id=input
then nothing more
When ls in the grafana folder, it showed
Service has entered the failed state with result exit-code.’ grafana-8.1.1-1.x86_64.rpm
grafana-server
Grafana-server Init Failed: Could not find config defaults, make sure homepath command line parameter is set or working directory is homepath