Grafana will not start after update from 3.1 to 4.3

[gps1539]

On centos 7 we ran ‘yum update grafana’ to update from 3.1 to 4.3. After the update grafana will not start.

systemctl status grafana-server
● grafana-server.service - Grafana instance
Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Tue 2017-05-23 19:59:03 PDT; 2s ago
Docs: http://docs.grafana.org
Process: 1704 ExecStart=/usr/sbin/grafana-server --config=${CONF_FILE} --pidfile=${PID_FILE} cfg:default.paths.logs=${LOG_DIR} cfg:default.paths.data=${DATA_DIR} cfg:default.paths.plugins=${PLUGINS_DIR} (code=exited, status=1/FAILURE)
Main PID: 1704 (code=exited, status=1/FAILURE)

May 23 19:59:02 test1 systemd[1]: grafana-server.service: main process exited, …URE
May 23 19:59:02 test1 systemd[1]: Unit grafana-server.service entered failed state.
May 23 19:59:02 test1 systemd[1]: grafana-server.service failed.
May 23 19:59:03 test1 systemd[1]: grafana-server.service holdoff time over, sch…rt.
May 23 19:59:03 test1 systemd[1]: start request repeated too quickly for grafan…ice
May 23 19:59:03 test1 systemd[1]: Failed to start Grafana instance.
May 23 19:59:03 test1 systemd[1]: Unit grafana-server.service entered failed state.
May 23 19:59:03 test1 systemd[1]: grafana-server.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

Is there a step we need to do to get grafana to run again?, thanks in advance

[torkel]

Can you check the grafana server log? /var/log/grafana/grafana.log

[gps1539]

@torkel

/var/log/grafana/grafana.log

Thanks for the reply

cat /var/log/grafana/grafana.log
t=2016-10-25T14:22:32-0700 lvl=info msg=“Received signal [terminated]. shutting down”
t=2017-05-23T19:45:11-0700 lvl=info msg=“Starting Grafana” logger=main version=3.1.1 commit=a4d2708 compiled=2016-08-01T03:20:16-0700
t=2017-05-23T19:45:11-0700 lvl=info msg=“Config loaded from” logger=settings file=/usr/share/grafana/conf/defaults.ini
t=2017-05-23T19:45:11-0700 lvl=info msg=“Config loaded from” logger=settings file=/etc/grafana/grafana.ini
t=2017-05-23T19:45:11-0700 lvl=info msg=“Config overriden from command line” logger=settings arg=“default.paths.data=/var/lib/grafana”
t=2017-05-23T19:45:11-0700 lvl=info msg=“Config overriden from command line” logger=settings arg=“default.paths.logs=/var/log/grafana”
t=2017-05-23T19:45:11-0700 lvl=info msg=“Config overriden from command line” logger=settings arg=“default.paths.plugins=/var/lib/grafana/plugins”
t=2017-05-23T19:45:11-0700 lvl=info msg=“Path Home” logger=settings path=/usr/share/grafana
t=2017-05-23T19:45:11-0700 lvl=info msg=“Path Data” logger=settings path=/var/lib/grafana
t=2017-05-23T19:45:11-0700 lvl=info msg=“Path Logs” logger=settings path=/var/log/grafana
t=2017-05-23T19:45:11-0700 lvl=info msg=“Path Plugins” logger=settings path=/var/lib/grafana/plugins
t=2017-05-23T19:45:11-0700 lvl=info msg=“Initializing DB” logger=sqlstore dbtype=sqlite3
t=2017-05-23T19:45:11-0700 lvl=info msg=“Starting DB migration” logger=migrator
t=2017-05-23T19:45:11-0700 lvl=info msg=“Starting plugin search” logger=plugins
t=2017-05-23T19:45:11-0700 lvl=info msg=“Server Listening” logger=server address=0.0.0.0:3000 protocol=http subUrl=
t=2017-05-23T19:45:53-0700 lvl=info msg=“Received signal [terminated]. shutting down”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Starting Grafana” logger=main version=4.3.1 commit=befc15c compiled=2017-05-23T06:50:22-0700
t=2017-05-23T19:46:43-0700 lvl=info msg=“Config loaded from” logger=settings file=/usr/share/grafana/conf/defaults.ini
t=2017-05-23T19:46:43-0700 lvl=info msg=“Config loaded from” logger=settings file=/etc/grafana/grafana.ini
t=2017-05-23T19:46:43-0700 lvl=info msg=“Config overriden from command line” logger=settings arg=“default.paths.data=/var/lib/grafana”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Config overriden from command line” logger=settings arg=“default.paths.logs=/var/log/grafana”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Config overriden from command line” logger=settings arg=“default.paths.plugins=/var/lib/grafana/plugins”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Path Home” logger=settings path=/usr/share/grafana
t=2017-05-23T19:46:43-0700 lvl=info msg=“Path Data” logger=settings path=/var/lib/grafana
t=2017-05-23T19:46:43-0700 lvl=info msg=“Path Logs” logger=settings path=/var/log/grafana
t=2017-05-23T19:46:43-0700 lvl=info msg=“Path Plugins” logger=settings path=/var/lib/grafana/plugins
t=2017-05-23T19:46:43-0700 lvl=info msg=“Initializing DB” logger=sqlstore dbtype=sqlite3
t=2017-05-23T19:46:43-0700 lvl=info msg=“Starting DB migration” logger=migrator
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Add column help_flags1 to user table”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Update user table charset”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Update temp_user table charset”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Update org table charset”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Update org_user table charset”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Add index for dashboard_id in dashboard_tag”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Update dashboard table charset”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Update dashboard_tag table charset”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Add secure json data column”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Update data_source table charset”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Update api_key table charset”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Update dashboard_snapshot table charset”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Update quota table charset”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Update plugin_setting table charset”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Update playlist table charset”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Update playlist_item table charset”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Update preferences table charset”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“create alert table v1”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id="add index alert org_id & id "
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“add index alert state”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“add index alert dashboard_id”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“create alert_notification table v1”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Add column is_default”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“add index alert_notification org_id & name”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Update alert table charset”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Update alert_notification table charset”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Drop old annotation table v4”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“create annotation table v5”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“add index annotation 0 v3”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“add index annotation 1 v3”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“add index annotation 2 v3”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“add index annotation 3 v3”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“add index annotation 4 v3”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Update annotation table charset”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“Add column region_id to annotation table”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Executing migration” logger=migrator id=“create test_data table”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Starting plugin search” logger=plugins
t=2017-05-23T19:46:43-0700 lvl=info msg=“Initializing Stream Manager”
t=2017-05-23T19:46:43-0700 lvl=info msg=“Initializing HTTP Server” logger=http.server address=0.0.0.0:3000 protocol=http subUrl= socket=
t=2017-05-23T19:46:43-0700 lvl=info msg=“Initializing Alerting” logger=alerting.engine
t=2017-05-23T19:46:43-0700 lvl=info msg=“Initializing CleanUpService” logger=cleanup
t=2017-05-23T19:47:02-0700 lvl=info msg=“Shutdown started” logger=server code=0 reason=“system signal: terminated”
t=2017-05-23T19:47:02-0700 lvl=info msg=“stopped http server” logger=http.server
t=2017-05-23T19:47:02-0700 lvl=crit msg=“Failed to parse /etc/grafana/grafana.ini, open /etc/grafana/grafana.ini: no such file or directory%!(EXTRA interface {}=)”
t=2017-05-23T19:47:02-0700 lvl=crit msg=“Failed to parse /etc/grafana/grafana.ini, open /etc/grafana/grafana.ini: no such file or directory%!(EXTRA interface {}=)”
t=2017-05-23T19:47:02-0700 lvl=crit msg=“Failed to parse /etc/grafana/grafana.ini, open /etc/grafana/grafana.ini: no such file or directory%!(EXTRA interface {}=)”
t=2017-05-23T19:47:03-0700 lvl=crit msg=“Failed to parse /etc/grafana/grafana.ini, open /etc/grafana/grafana.ini: no such file or directory%!(EXTRA interface {}=)”
t=2017-05-23T19:47:03-0700 lvl=crit msg=“Failed to parse /etc/grafana/grafana.ini, open /etc/grafana/grafana.ini: no such file or directory%!(EXTRA interface {}=)”
t=2017-05-23T19:59:01-0700 lvl=crit msg=“Failed to parse /etc/grafana/grafana.ini, open /etc/grafana/grafana.ini: no such file or directory%!(EXTRA interface {}=)”
t=2017-05-23T19:59:02-0700 lvl=crit msg=“Failed to parse /etc/grafana/grafana.ini, open /etc/grafana/grafana.ini: no such file or directory%!(EXTRA interface {}=)”
t=2017-05-23T19:59:02-0700 lvl=crit msg=“Failed to parse /etc/grafana/grafana.ini, open /etc/grafana/grafana.ini: no such file or directory%!(EXTRA interface {}=)”
t=2017-05-23T19:59:02-0700 lvl=crit msg=“Failed to parse /etc/grafana/grafana.ini, open /etc/grafana/grafana.ini: no such file or directory%!(EXTRA interface {}=)”
t=2017-05-23T19:59:02-0700 lvl=crit msg=“Failed to parse /etc/grafana/grafana.ini, open /etc/grafana/grafana.ini: no such file or directory%!(EXTRA interface {}=)”

---

the file /etc/grafana/grafana.ini does not exist

[torkel]

Do you have any files in /etc/grafana/?

[gps1539]

Thanks
There are no files in /etc/grafana/
and find / -name grafana.ini
finds nothing

---

We restored from a snapshot so we’re back on 3.1 and retried the update. The config file is removed.

Before
ls /etc/grafana/
grafana.ini ldap.toml

yum install grafana
Loaded plugins: fastestmirror
base | 3.6 kB 00:00
centosplus | 3.4 kB 00:00
elrepo | 2.9 kB 00:00
epel/x86_64/metalink | 13 kB 00:00
epel | 4.3 kB 00:00
extras | 3.4 kB 00:00
grafana/x86_64/signature | 836 B 00:00
grafana/x86_64/signature | 1.0 kB 00:00 !!!
influxdb | 2.4 kB 00:00
updates | 3.4 kB 00:00
(1/10): base/7/x86_64/group_gz | 155 kB 00:00
(2/10): elrepo/primary_db | 413 kB 00:00
(3/10): epel/x86_64/group_gz | 170 kB 00:00
(4/10): extras/7/x86_64/primary_db | 151 kB 00:00
(5/10): epel/x86_64/updateinfo | 799 kB 00:01
(6/10): influxdb/7/x86_64/primary_db | 18 kB 00:00
(7/10): centosplus/7/x86_64/primary_db | 2.2 MB 00:01
(8/10): epel/x86_64/primary_db | 4.7 MB 00:01
(9/10): base/7/x86_64/primary_db | 5.6 MB 00:01
(10/10): updates/7/x86_64/primary_db | 4.8 MB 00:00
grafana/x86_64/primary | 3.1 kB 00:00
Determining fastest mirrors

• base: centos-distro.cavecreek.net
• centosplus: mirrors.maine.edu
• elrepo: elrepo.reloumirrors.net
• epel: mirror.oss.ou.edu
• extras: mirrors.maine.edu
• updates: mirror.hmc.edu
  grafana 23/23
  Resolving Dependencies
  –> Running transaction check
  —> Package grafana.x86_64 0:3.1.1-1470047149 will be updated
  —> Package grafana.x86_64 0:4.3.1-1 will be an update
  –> Finished Dependency Resolution

Dependencies Resolved

================================================================================
Package Arch Version Repository Size

Updating:
grafana x86_64 4.3.1-1 grafana 45 M

Transaction Summary

Upgrade 1 Package

Total download size: 45 M
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
warning: /var/cache/yum/x86_64/7/grafana/packages/grafana-4.3.1-1.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 24098cb6: NOKEY
Public key for grafana-4.3.1-1.x86_64.rpm is not installed
grafana-4.3.1-1.x86_64.rpm | 45 MB 00:05

(had to remove output here as the forum blocks links, just urls for package and keys)

Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : grafana-4.3.1-1.x86_64 1/2
Cleanup : grafana-3.1.1-1470047149.x86_64 2/2
POSTTRANS: Running script
POSTTRANS: Config file not found
Verifying : grafana-4.3.1-1.x86_64 1/2
Verifying : grafana-3.1.1-1470047149.x86_64 2/2

Updated:
grafana.x86_64 0:4.3.1-1

Complete!
[root@test1 ~]# ls /etc/grafana/
[root@test1 ~]#

[gps1539]

After restoring /etc/grafana/grafana.ini and /etc/grafana/ldap.toml grafana starts again. Looks like something eats them during the update.

[captjak]

I just updated from 3.1.1 to 4.3.1 on CentOS 7 and my config file has gone as well. Nothing in /etc/grafana.

Looks like I’m going to have to rebuild it…

Perhaps there should be a mention to back up the config file as well when upgrading, not just the DB (which I did back up, but wasn’t affected by the upgrade)?

[torkel]

Did you check in /etc/grafana/ any files there? Very strange that this happens two a few people, being trying to replicate without success

[captjak]

Yeah, I checked /etc/grafana and it’s empty. Just like the last user, all I needed was that config file and it started up again with no problem, so we’re all good now.

Interesting though that it’s specifically from 3.1 to 4.3.1 in both our cases. Have you tried to reproduce with those versions on CentOS?

[captjak]

And FWIW, when I tried to upgrade, I ran yum localinstall grafana-4.3.1-1.x86_64.rpm and it ran as an update (at least that’s what yum told me).

And I just downloaded the .rpm two hours ago, so it’s a recent file.

[torkel]

Tested this again on CentOS 7, installed first Grafana 3.1, modified config file them did yum localinstall grafana-4.3.1-1.x86_64.rpm

Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Uppdaterar  : grafana-4.3.1-1.x86_64                                                                                                                                                                                                    1/2 
  Rensar upp  : grafana-3.1.1-1470047149.x86_64                                                                                                                                                                                           2/2 
varning: /etc/grafana/grafana.ini sparades som /etc/grafana/grafana.ini.rpmsave
POSTTRANS: Running script
POSTTRANS: Config file not found
POSTTRANS: /etc/grafana/grafana.ini.rpmsave config file found.
POSTTRANS: /etc/grafana/grafana.ini restored
POSTTRANS: Restoring config file permissions
  Verifierar  : grafana-4.3.1-1.x86_64                                                                                                                                                                                                    1/2 
  Verifierar  : grafana-3.1.1-1470047149.x86_64                                                                                                                                                                                           2/2 

The post install script is running and restoring the config file, so after the install my config file is still there where it should be. Do you have logs from when you upgraded?