My grafana-server is contastantly restarting and I have no idea what is causing the issue.
Restarting server does help, but after upgrade grafana or restarting grafana the issue appears.
Here’s the logs:
logger=settings t=2022-04-23T23:09:56.75+0200 lvl=info msg=“Path Logs” path=/var/log/grafana
logger=settings t=2022-04-23T23:09:56.75+0200 lvl=info msg=“Path Plugins” path=/var/lib/grafana/plugins
logger=settings t=2022-04-23T23:09:56.75+0200 lvl=info msg=“Path Provisioning” path=/etc/grafana/provisioning
logger=settings t=2022-04-23T23:09:56.75+0200 lvl=info msg=“App mode production”
logger=sqlstore t=2022-04-23T23:09:56.75+0200 lvl=info msg=“Connecting to DB” dbtype=sqlite3
logger=migrator t=2022-04-23T23:09:56.84+0200 lvl=info msg=“Starting DB migrations”
logger=migrator t=2022-04-23T23:09:56.88+0200 lvl=info msg=“migrations completed” performed=0 skipped=381 duration=3.899093ms
logger=plugin.manager t=2022-04-23T23:09:57.13+0200 lvl=info msg=“Plugin registered” pluginId=input
logger=query_data t=2022-04-23T23:09:57.13+0200 lvl=info msg=“Query Service initialization”
logger=live.push_http t=2022-04-23T23:09:57.15+0200 lvl=info msg=“Live Push Gateway initialization”
logger=settings t=2022-04-23T23:10:08.25+0200 lvl=info msg=“Starting Grafana” version=8.4.7 commit=23bf3ef043 branch=HEAD compiled=2022-04-19T18:05:49+02:00
logger=settings t=2022-04-23T23:10:08.25+0200 lvl=info msg=“The state of unified alerting is still not defined. The decision will be made during as we run the database migrations”
logger=settings t=2022-04-23T23:10:08.25+0200 lvl=warn msg=“falling back to legacy setting of ‘min_interval_seconds’; please use the configuration option in the unified_alerting section if Grafana 8 alerts are enabled.”
logger=settings t=2022-04-23T23:10:08.25+0200 lvl=info msg=“Config loaded from” file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2022-04-23T23:10:08.25+0200 lvl=info msg=“Config loaded from” file=/etc/grafana/grafana.ini
logger=settings t=2022-04-23T23:10:08.25+0200 lvl=info msg=“Config overridden from command line” arg=“default.paths.data=/var/lib/grafana”
logger=settings t=2022-04-23T23:10:08.25+0200 lvl=info msg=“Config overridden from command line” arg=“default.paths.logs=/var/log/grafana”
logger=settings t=2022-04-23T23:10:08.25+0200 lvl=info msg=“Config overridden from command line” arg=“default.paths.plugins=/var/lib/grafana/plugins”
logger=settings t=2022-04-23T23:10:08.25+0200 lvl=info msg=“Config overridden from command line” arg=“default.paths.provisioning=/etc/grafana/provisioning”
logger=settings t=2022-04-23T23:10:08.25+0200 lvl=info msg=“Path Home” path=/usr/share/grafana
logger=settings t=2022-04-23T23:10:08.25+0200 lvl=info msg=“Path Data” path=/var/lib/grafana
logger=settings t=2022-04-23T23:10:08.25+0200 lvl=info msg=“Path Logs” path=/var/log/grafana
logger=settings t=2022-04-23T23:10:08.25+0200 lvl=info msg=“Path Plugins” path=/var/lib/grafana/plugins
logger=settings t=2022-04-23T23:10:08.25+0200 lvl=info msg=“Path Provisioning” path=/etc/grafana/provisioning
logger=settings t=2022-04-23T23:10:08.25+0200 lvl=info msg=“App mode production”
logger=sqlstore t=2022-04-23T23:10:08.25+0200 lvl=info msg=“Connecting to DB” dbtype=sqlite3
logger=migrator t=2022-04-23T23:10:08.34+0200 lvl=info msg=“Starting DB migrations”
logger=migrator t=2022-04-23T23:10:08.37+0200 lvl=info msg=“migrations completed” performed=0 skipped=381 duration=3.621429ms
logger=plugin.manager t=2022-04-23T23:10:08.59+0200 lvl=info msg=“Plugin registered” pluginId=input
logger=query_data t=2022-04-23T23:10:08.59+0200 lvl=info msg=“Query Service initialization”
logger=live.push_http t=2022-04-23T23:10:08.62+0200 lvl=info msg=“Live Push Gateway initialization”
logger=settings t=2022-04-23T23:10:19.77+0200 lvl=info msg=“Starting Grafana” version=8.4.7 commit=23bf3ef043 branch=HEAD compiled=2022-04-19T18:05:49+02:00
logger=settings t=2022-04-23T23:10:19.77+0200 lvl=info msg=“The state of unified alerting is still not defined. The decision will be made during as we run the database migrations”
logger=settings t=2022-04-23T23:10:19.77+0200 lvl=warn msg=“falling back to legacy setting of ‘min_interval_seconds’; please use the configuration option in the unified_alerting section if Grafana 8 alerts are enabled.”
logger=settings t=2022-04-23T23:10:19.77+0200 lvl=info msg=“Config loaded from” file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2022-04-23T23:10:19.77+0200 lvl=info msg=“Config loaded from” file=/etc/grafana/grafana.ini
logger=settings t=2022-04-23T23:10:19.77+0200 lvl=info msg=“Config overridden from command line” arg=“default.paths.data=/var/lib/grafana”
logger=settings t=2022-04-23T23:10:19.77+0200 lvl=info msg=“Config overridden from command line” arg=“default.paths.logs=/var/log/grafana”
logger=settings t=2022-04-23T23:10:19.77+0200 lvl=info msg=“Config overridden from command line” arg=“default.paths.plugins=/var/lib/grafana/plugins”
logger=settings t=2022-04-23T23:10:19.77+0200 lvl=info msg=“Config overridden from command line” arg=“default.paths.provisioning=/etc/grafana/provisioning”
logger=settings t=2022-04-23T23:10:19.77+0200 lvl=info msg=“Path Home” path=/usr/share/grafana
logger=settings t=2022-04-23T23:10:19.77+0200 lvl=info msg=“Path Data” path=/var/lib/grafana
logger=settings t=2022-04-23T23:10:19.77+0200 lvl=info msg=“Path Logs” path=/var/log/grafana
logger=settings t=2022-04-23T23:10:19.77+0200 lvl=info msg=“Path Plugins” path=/var/lib/grafana/plugins
logger=settings t=2022-04-23T23:10:19.77+0200 lvl=info msg=“Path Provisioning” path=/etc/grafana/provisioning
logger=settings t=2022-04-23T23:10:19.77+0200 lvl=info msg=“App mode production”
logger=sqlstore t=2022-04-23T23:10:19.78+0200 lvl=info msg=“Connecting to DB” dbtype=sqlite3
logger=migrator t=2022-04-23T23:10:19.88+0200 lvl=info msg=“Starting DB migrations”
logger=migrator t=2022-04-23T23:10:19.91+0200 lvl=info msg=“migrations completed” performed=0 skipped=381 duration=3.443968ms
logger=plugin.manager t=2022-04-23T23:10:20.15+0200 lvl=info msg=“Plugin registered” pluginId=input
logger=query_data t=2022-04-23T23:10:20.15+0200 lvl=info msg=“Query Service initialization”
logger=live.push_http t=2022-04-23T23:10:20.18+0200 lvl=info msg=“Live Push Gateway initialization”
logger=settings t=2022-04-23T23:10:31.21+0200 lvl=info msg=“Starting Grafana” version=8.4.7 commit=23bf3ef043 branch=HEAD compiled=2022-04-19T18:05:49+02:00
logger=settings t=2022-04-23T23:10:31.21+0200 lvl=info msg=“The state of unified alerting is still not defined. The decision will be made during as we run the database migrations”
logger=settings t=2022-04-23T23:10:31.21+0200 lvl=warn msg=“falling back to legacy setting of ‘min_interval_seconds’; please use the configuration option in the unified_alerting section if Grafana 8 alerts are enabled.”
logger=settings t=2022-04-23T23:10:31.21+0200 lvl=info msg=“Config loaded from” file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2022-04-23T23:10:31.21+0200 lvl=info msg=“Config loaded from” file=/etc/grafana/grafana.ini
logger=settings t=2022-04-23T23:10:31.21+0200 lvl=info msg=“Config overridden from command line” arg=“default.paths.data=/var/lib/grafana”
logger=settings t=2022-04-23T23:10:31.21+0200 lvl=info msg=“Config overridden from command line” arg=“default.paths.logs=/var/log/grafana”
logger=settings t=2022-04-23T23:10:31.21+0200 lvl=info msg=“Config overridden from command line” arg=“default.paths.plugins=/var/lib/grafana/plugins”
logger=settings t=2022-04-23T23:10:31.21+0200 lvl=info msg=“Config overridden from command line” arg=“default.paths.provisioning=/etc/grafana/provisioning”
logger=settings t=2022-04-23T23:10:31.21+0200 lvl=info msg=“Path Home” path=/usr/share/grafana
logger=settings t=2022-04-23T23:10:31.21+0200 lvl=info msg=“Path Data” path=/var/lib/grafana
logger=settings t=2022-04-23T23:10:31.21+0200 lvl=info msg=“Path Logs” path=/var/log/grafana
logger=settings t=2022-04-23T23:10:31.21+0200 lvl=info msg=“Path Plugins” path=/var/lib/grafana/plugins
logger=settings t=2022-04-23T23:10:31.21+0200 lvl=info msg=“Path Provisioning” path=/etc/grafana/provisioning
logger=settings t=2022-04-23T23:10:31.21+0200 lvl=info msg=“App mode production”
logger=sqlstore t=2022-04-23T23:10:31.21+0200 lvl=info msg=“Connecting to DB” dbtype=sqlite3
logger=migrator t=2022-04-23T23:10:31.31+0200 lvl=info msg=“Starting DB migrations”
logger=migrator t=2022-04-23T23:10:31.34+0200 lvl=info msg=“migrations completed” performed=0 skipped=381 duration=3.599003ms
logger=plugin.manager t=2022-04-23T23:10:31.56+0200 lvl=info msg=“Plugin registered” pluginId=input
logger=query_data t=2022-04-23T23:10:31.57+0200 lvl=info msg=“Query Service initialization”
logger=live.push_http t=2022-04-23T23:10:31.59+0200 lvl=info msg=“Live Push Gateway initialization”
logger=settings t=2022-04-23T23:10:43.01+0200 lvl=info msg=“Starting Grafana” version=8.4.7 commit=23bf3ef043 branch=HEAD compiled=2022-04-19T18:05:49+02:00
logger=settings t=2022-04-23T23:10:43.01+0200 lvl=info msg=“The state of unified alerting is still not defined. The decision will be made during as we run the database migrations”
logger=settings t=2022-04-23T23:10:43.01+0200 lvl=warn msg=“falling back to legacy setting of ‘min_interval_seconds’; please use the configuration option in the unified_alerting section if Grafana 8 alerts are enabled.”
logger=settings t=2022-04-23T23:10:43.01+0200 lvl=info msg=“Config loaded from” file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2022-04-23T23:10:43.01+0200 lvl=info msg=“Config loaded from” file=/etc/grafana/grafana.ini
logger=settings t=2022-04-23T23:10:43.01+0200 lvl=info msg=“Config overridden from command line” arg=“default.paths.data=/var/lib/grafana”
logger=settings t=2022-04-23T23:10:43.01+0200 lvl=info msg=“Config overridden from command line” arg=“default.paths.logs=/var/log/grafana”
logger=settings t=2022-04-23T23:10:43.01+0200 lvl=info msg=“Config overridden from command line” arg=“default.paths.plugins=/var/lib/grafana/plugins”
logger=settings t=2022-04-23T23:10:43.01+0200 lvl=info msg=“Config overridden from command line” arg=“default.paths.provisioning=/etc/grafana/provisioning”
logger=settings t=2022-04-23T23:10:43.01+0200 lvl=info msg=“Path Home” path=/usr/share/grafana
logger=settings t=2022-04-23T23:10:43.01+0200 lvl=info msg=“Path Data” path=/var/lib/grafana
logger=settings t=2022-04-23T23:10:43.01+0200 lvl=info msg=“Path Logs” path=/var/log/grafana
logger=settings t=2022-04-23T23:10:43.01+0200 lvl=info msg=“Path Plugins” path=/var/lib/grafana/plugins
logger=settings t=2022-04-23T23:10:43.01+0200 lvl=info msg=“Path Provisioning” path=/etc/grafana/provisioning
logger=settings t=2022-04-23T23:10:43.01+0200 lvl=info msg=“App mode production”
logger=sqlstore t=2022-04-23T23:10:43.01+0200 lvl=info msg=“Connecting to DB” dbtype=sqlite3
logger=migrator t=2022-04-23T23:10:43.1+0200 lvl=info msg=“Starting DB migrations”
logger=migrator t=2022-04-23T23:10:43.14+0200 lvl=info msg=“migrations completed” performed=0 skipped=381 duration=3.70378ms
logger=plugin.manager t=2022-04-23T23:10:43.35+0200 lvl=info msg=“Plugin registered” pluginId=input
logger=query_data t=2022-04-23T23:10:43.36+0200 lvl=info msg=“Query Service initialization”
logger=live.push_http t=2022-04-23T23:10:43.38+0200 lvl=info msg=“Live Push Gateway initialization”
I’m using version=8.4.7 in rasberri pi 8Gb.
howdy,
can you try increasing the verbosity of the Grafana server logs to debug and note any errors. For printing to console, set the console logs to debug as well.
Hello,
so the issue heppens everytime I upgrade grafana to newer version.
Here is the debug logs: logger=migrator t=2022-07-19T08:58:19.564235915+02:00 level=info msg="Starting D - Pastebin.com
Grafana restart about every 10s and after that there is an instance with for example more than 100s running:
root@raspberrypi:/var/log/grafana# systemctl status grafana-server
● grafana-server.service - Grafana instance
Loaded: loaded (/lib/systemd/system/grafana-server.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-07-19 09:04:29 CEST; 9s ago
Main PID: 32400 (grafana-server)
Tasks: 8 (limit: 4915)
● grafana-server.service - Grafana instance
Loaded: loaded (/lib/systemd/system/grafana-server.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-07-19 09:04:40 CEST; 167ms ago
Main PID: 32426 (grafana-server)
Tasks: 6 (limit: 4915)
CPU: 164ms
grafana-server.service - Grafana instance
Loaded: loaded (/lib/systemd/system/grafana-server.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-07-19 09:08:08 CEST; 1s ago
Main PID: 32754 (grafana-server)
Tasks: 9 (limit: 4915)
grafana-server.service - Grafana instance
Loaded: loaded (/lib/systemd/system/grafana-server.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-07-19 09:10:15 CEST; 14ms ago
Main PID: 776 ((a-server))
Tasks: 1 (limit: 4915)
CPU: 9ms
CGroup: /system.slice/grafana-server.service
└─776 (a-server)
Jul 19 09:10:15 raspberrypi systemd[1]: Started Grafana instance.
root@raspberrypi:/var/log/grafana# systemctl status grafana-server
● grafana-server.service - Grafana instance
Loaded: loaded (/lib/systemd/system/grafana-server.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-07-19 09:10:15 CEST; 3s ago
Main PID: 776 (grafana-server)
Tasks: 8 (limit: 4915)
CPU: 1.140s
CGroup: /system.slice/grafana-server.service
└─776 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/run/grafana/grafana-server.pid --packaging=deb cfg:default.paths.logs=/var/log/grafana cfg:default.paths.data=/var/lib/gr>
Jul 19 09:10:15 raspberrypi systemd[1]: Started Grafana instance.
output of grafana-server is:
Grafana server is running with elevated privileges. This is not recommended
Grafana-server Init Failed: Could not find config defaults, make sure homepath command line parameter is set or working directory is homepath
so is the issue with config file location?