Grafana server starting to failed

Grafana Dashboards
1

My Grafana is starting to failed, As it was failing I upgraded it without taking backup Here are some details of files and error -
Config files:
/usr/share/grafana/conf/defaults.ini
/etc/grafana/grafana.ini
Command lines overrides:
data=/var/lib/grafana (plugins,csv,alerting files)
logs=/var/log/grafana

systemctl start grafana-server
Job for grafana-server.service failed because the control process exited with error code. See "systemctl status grafana-server.service" and "journalctl -xe" for details.

And Status systemctl status grafana-server
● grafana-server.service - Grafana instance
   Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Tue 2024-07-30 05:37:25 UTC; 6min ago
     Docs: http://docs.grafana.org
  Process: 1341 ExecStart=/usr/share/grafana/bin/grafana server --config=${CONF_FILE} --pidfile=${PID_FILE_DIR}/grafana-server.pid --packaging=rpm cfg:default.paths.logs=${LOG_DIR} cfg:default.paths.data=${DATA_DIR} cfg:default.paths.plugins=${PLUGINS_DIR} cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR} (code=exited, status=1/FAILURE)
 Main PID: 1341 (code=exited, status=1/FAILURE)

Jul 30 05:37:25 analytics-app-vm systemd[1]: Failed to start Grafana instance.
Jul 30 05:37:25 analytics-app-vm systemd[1]: Unit grafana-server.service entered failed state.
Jul 30 05:37:25 analytics-app-vm systemd[1]: grafana-server.service failed.
Jul 30 05:37:25 analytics-app-vm systemd[1]: grafana-server.service holdoff time over, scheduling restart.
Jul 30 05:37:25 analytics-app-vm systemd[1]: Stopped Grafana instance.
Jul 30 05:37:25 analytics-app-vm systemd[1]: start request repeated too quickly for grafana-server.service
Jul 30 05:37:25 analytics-app-vm systemd[1]: Failed to start Grafana instance.
Jul 30 05:37:25 analytics-app-vm systemd[1]: Unit grafana-server.service entered failed state.
Jul 30 05:37:25 analytics-app-vm systemd[1]: grafana-server.service failed.
2

Check the logs of grafana (not grafana service).

3

I could only find logs here var/log/grafana/

logger=remotecache.database t=2024-07-29T03:08:54.570915254Z level=error msg="failed to run garbage collect" error="dial tcp 127.0.0.1:3306: connect: connection refused"
logger=cleanup t=2024-07-29T03:08:54.571418316Z level=error msg="Failed to delete expired snapshots" error="dial tcp 127.0.0.1:3306: connect: connection refused"
logger=cleanup t=2024-07-29T03:08:54.571642321Z level=error msg="Failed to delete expired dashboard versions" error="dial tcp 127.0.0.1:3306: connect: connection refused"
logger=cleanup t=2024-07-29T03:08:54.571844123Z level=error msg="Failed to delete expired images" error="dial tcp 127.0.0.1:3306: connect: connection refused"
logger=cleanup t=2024-07-29T03:08:54.572223771Z level=error msg="Problem expiring user invites" error="dial tcp 127.0.0.1:3306: connect: connection refused"
logger=cleanup t=2024-07-29T03:08:54.572462742Z level=error msg="Problem deleting stale short urls" error="dial tcp 127.0.0.1:3306: connect: connection refused"
logger=cleanup t=2024-07-29T03:08:54.57266656Z level=error msg="Problem deleting stale query history" error="dial tcp 127.0.0.1:3306: connect: connection refused"
logger=cleanup t=2024-07-29T03:08:54.572840098Z level=error msg="Problem with enforcing row limit for query_history" error="dial tcp 127.0.0.1:3306: connect: connection refused"
logger=cleanup t=2024-07-29T03:08:54.573063587Z level=error msg="Problem with enforcing row limit for query_history_star" error="dial tcp 127.0.0.1:3306: connect: connection refused"
logger=cleanup t=2024-07-29T03:08:54.573199308Z level=info msg="Completed cleanup jobs" duration=2.659244ms
logger=grafana.update.checker t=2024-07-29T03:08:54.667290198Z level=info msg="Update check succeeded" duration

logger=ngalert.scheduler t=2024-07-29T03:09:00.000555247Z level=error msg="Failed to update alert rules" error="failed to get alert rules: failed to fetch alert rules: dial tcp 127.0.0.1:3306: connect: connection refused"
4

Milions errors about database - it’s not able to connect to db. Fix that and one problem solved.
Please format logs.

5

To resolve the Grafana startup issue, first check the logs for specific error messages using journalctl -xe and systemctl status grafana-server. Verify the integrity and correctness of the configuration files located at /usr/share/grafana/conf/defaults.ini and /etc/grafana/grafana.ini. Ensure that Grafana has the correct permissions for directories /var/lib/grafana and /var/log/grafana. If you encounter issues with backups, temporarily remove or rename the existing backup files. Restart the Grafana service using sudo systemctl restart grafana-server. If the issue persists, consider reinstalling Grafana.

Shanker Agarwal
(Grafana Architect)
shanker.agarwal@dataabsolute.com