Using Grafana OSS on Windows 2008r2. I now have Grafana 6.7.3. I’m looking at upgrading to V 11.0 OSS.
It is working fine, but I’m seeing the system hang when it attempts to shut down or restart and the shutdown of Grafana is the last thing in the system log before I have to force a restart with a power cycle:
Before I get into debugging this issue, I want to have the latest version.
Are there any tips or pointers I should keep in mind when upgrading from 6 to 11? I assume it can be done in one step? Above and beyond following the instructions here ?
These instructions for Windows say “In your current installation of Grafana, save your custom configuration changes to a file named <grafana_install_dir>/conf/custom.ini.”
My config folder C:\Program Files\GrafanaLabs\grafana\conf contains defaults.ini and sample.ini. There is no Custom.ini. It is unclear how to “save your custom configuration”
The instructions then say “Extract the contents of the Windows Binary package to the location in which you installed Grafana.” I assume it would be necessary to stop the service (path to c:\Program Files\GrafanaLabs\nssm.exe) from the Services manager before overwriting the binary files?
Background on shutdown hang: From System Log: (shutdown initiated at 3:49. Noticed failure and power cycled at 4:34.
Level Date and Time Source Event ID Task Category
Information 5/26/2024 4:34:36 PM Microsoft-Windows-User Profiles Service 1531 None "The User Profile Service has started successfully.
Information 5/26/2024 3:49:48 PM nssm 1011 None Killing process C:\Program Files\GrafanaLabs\grafana\bin\grafana-server.exe because service Grafana is stopping.
Information 5/26/2024 3:49:48 PM nssm 1040 None Service Grafana received SHUTDOWN control, which will be handled.
To confirm, I download the Windows Binaries for versions 7 through 11. I pick the last of each major version series 7.5.17, 8.5.27, etc. (presumably the most stable?)
Since I do not have a custom.ini, there is no file that needs to be saved in the conf directory?
So the process is like this (in python pseudo code)
for x in range (7, 12):
stop Grafana Service
extract binaries for version x into "C:\Program Files\GrafanaLabs\grafana" and overwrite
start Grafana service
test Dashboard
In case of failure, extract prior version and then ???
I would not overwrite existing grafana folder. This is what I do personally
I have a folder in C drive named LGTM
I download the grafana zip file for windows and extract the zip file, copy pertinent folders from previous version to new version folder, run grafana-server.exe in bin folder in new version folder and verify all is working.
no files except the data folder where your plugins and grafana.db are if you are using sqllite as a backend for grafana.
I have a service that is set to automatic.
It runs C:\Program Files\GrafanaLabs\nssm.exe
So I would stop the service, create the folders for the multiple versions such as:
c:\temp\grafana-7-5-17
and extract grafana-7.5.17.windows-amd64.zip into that folder. Then copy the contents of C:\Program Files\GrafanaLabs\grafana\data (my current data) into the c:\temp\grafana-7-5-17 folder. Then run grafana-server.exe in c:\temp\grafana-7-5-17\bin folder
Iterate through versions, moving the \data folder to each new version’s folder
When all versions have been tested, copy the final test folder c:\temp\grafana-11-0-0 back to C:\Program Files\GrafanaLabs\grafana.
Then change the service back to automatic.
I copied \data to the V7 folder, and ran grafana-server.exe from \bin
That worked.
Then I moved \data to the V8 folder and ran grafana-server.exe from \bin
That worked
Then I moved \data to the V9 folder and ran grafana-server.exe from \bin
That fails with an exception, same as V11
I noticed that when the server ran, it logged "Executing migration " so apparently migration is necessary, but the migration to V9 is problematic?
Yes the prior server was always stopped. I had stopped the service before starting, and was running grafana-server.exe from the CLI as foreground, so I could not run the next version until I killed the prior one with ctrl-C.
Here’s the output from running from CLI after copying V8 files back to the normal execution location. C:\Program Files\GrafanaLabs\grafana
There are some errors but it does appear to work normally. (it looks like output is for ANSI terminal, not windows CMD)
C:\Program Files\GrafanaLabs\grafana\bin>grafana-server.exe
←[32mINFO ←[0m[06-01|12:55:04] Starting Grafana ←[32mlogger←[0m=settings ←[32mversion←[0m=8.5.27 ←[32mcommit←[0m=5f9e0ace7e ←[32mbranch←[0m=HEAD ←[32mcompil
ed←[0m=2023-06-07T05:18:00-0500
←[32mINFO ←[0m[06-01|12:55:04] The state of unified alerting is still not defined. The decision will be made during as we run the database migrations ←[32mlogger←[0m=settings
←[32mINFO ←[0m[06-01|12:55:04] Config loaded from ←[32mlogger←[0m=settings ←[32mfile←[0m="C:\\Program Files\\GrafanaLabs\\grafana/conf/defaults.ini"
←[32mINFO ←[0m[06-01|12:55:04] Path Home ←[32mlogger←[0m=settings ←[32mpath←[0m="C:\\Program Files\\GrafanaLabs\\grafana"
←[32mINFO ←[0m[06-01|12:55:04] Path Data ←[32mlogger←[0m=settings ←[32mpath←[0m="C:\\Program Files\\GrafanaLabs\\grafana\\data"
←[32mINFO ←[0m[06-01|12:55:04] Path Logs ←[32mlogger←[0m=settings ←[32mpath←[0m="C:\\Program Files\\GrafanaLabs\\grafana\\data\\log"
←[32mINFO ←[0m[06-01|12:55:04] Path Plugins ←[32mlogger←[0m=settings ←[32mpath←[0m="C:\\Program Files\\GrafanaLabs\\grafana\\data\\plugins"
←[32mINFO ←[0m[06-01|12:55:04] Path Provisioning ←[32mlogger←[0m=settings ←[32mpath←[0m="C:\\Program Files\\GrafanaLabs\\grafana\\conf\\provisioning"
←[32mINFO ←[0m[06-01|12:55:04] App mode production ←[32mlogger←[0m=settings
←[32mINFO ←[0m[06-01|12:55:04] Connecting to DB ←[32mlogger←[0m=sqlstore ←[32mdbtype←[0m=sqlite3
←[33mWARN ←[0m[06-01|12:55:04] SQLite database file has broader permissions than it should ←[33mlogger←[0m=sqlstore ←[33mpath←[0m="C:\\Program Files\\GrafanaLabs\\grafana\\data\\gr
afana.db" ←[33mmode←[0m=-rw-rw-rw- ←[33mexpected←[0m=-rw-r-----
←[32mINFO ←[0m[06-01|12:55:04] Starting DB migrations ←[32mlogger←[0m=migrator
←[32mINFO ←[0m[06-01|12:55:04] migrations completed ←[32mlogger←[0m=migrator ←[32mperformed←[0m=0 ←[32mskipped←[0m=392 ←[32mduration←[0m=1ms
←[33mWARN ←[0m[06-01|12:55:04] error loading storage ←[33mlogger←[0m=grafanaStorageLogger ←[33mprefix←[0m=public-static ←[33merr←[0m="open blob.Bucket: parse \"f
ile://C:\\\\Program Files\\\\GrafanaLabs\\\\grafana\\\\public\": invalid port \":\\\\Program Files\\\\GrafanaLabs\\\\grafana\\\\public\" after host"
←[31mEROR ←[0m[06-01|12:55:05] Could not start plugin ←[31mlogger←[0m=plugin.manager ←[31mpluginId←[0m=stackdriver ←[31merr←[0m="plugin stackdriver is already reg
istered"
←[32mINFO ←[0m[06-01|12:55:05] Query Service initialization ←[32mlogger←[0m=query_data
←[32mINFO ←[0m[06-01|12:55:05] Live Push Gateway initialization ←[32mlogger←[0m=live.push_http
←[32mINFO ←[0m[06-01|12:55:06] warming cache for startup ←[32mlogger←[0m=ngalert
←[32mINFO ←[0m[06-01|12:55:06] starting MultiOrg Alertmanager ←[32mlogger←[0m=ngalert.multiorg.alertmanager
←[32mINFO ←[0m[06-01|12:55:06] storage starting ←[32mlogger←[0m=grafanaStorageLogger
←[32mINFO ←[0m[06-01|12:55:06] HTTP Server Listen ←[32mlogger←[0m=http.server ←[32maddress←[0m=[::]:3000 ←[32mprotocol←[0m=http ←[32msubUrl←[0m= ←[32msocket←[
0m=
←[32mINFO ←[0m[06-01|12:55:50] Request Completed ←[32mlogger←[0m=context ←[32mtraceID←[0m=00000000000000000000000000000000 ←[32muserId←[0m=1 ←[32morgId←[0m=1
←[32muname←[0m=admin ←[32mmethod←[0m=GET ←[32mpath←[0m=/api/live/ws ←[32mstatus←[0m=0 ←[32mremote_addr←[0m=192.168.27.106 ←[32mtime_ms←[0m=6 ←[32mduration←[0m=6.0004ms ←[32msize←[
0m=0 ←[32mreferer←[0m= ←[32mtraceID←[0m=00000000000000000000000000000000
←[32mINFO ←[0m[06-01|12:55:50] Initialized channel handler ←[32mlogger←[0m=live ←[32mchannel←[0m=grafana/dashboard/uid/jZ5Ky1RGk ←[32maddress←[0m=grafana/dashboard/uid
/jZ5Ky1RGk
←[32mINFO ←[0m[06-01|12:55:56] Request Completed ←[32mlogger←[0m=context ←[32mtraceID←[0m=00000000000000000000000000000000 ←[32muserId←[0m=1 ←[32morgId←[0m=1
←[32muname←[0m=admin ←[32mmethod←[0m=GET ←[32mpath←[0m=/api/live/ws ←[32mstatus←[0m=0 ←[32mremote_addr←[0m=192.168.27.106 ←[32mtime_ms←[0m=13 ←[32mduration←[0m=13.0007ms ←[32msize
←[0m=0 ←[32mreferer←[0m= ←[32mtraceID←[0m=00000000000000000000000000000000
←[32mINFO ←[0m[06-01|12:56:33] Shutdown started ←[32mlogger←[0m=server ←[32mreason←[0m="System signal: interrupt"
←[32mINFO ←[0m[06-01|12:56:33] Closing tracing ←[32mlogger←[0m=tracing
←[31mEROR ←[0m[06-01|12:56:33] context canceled ←[31mlogger←[0m=ngalert
C:\Program Files\GrafanaLabs\grafana\bin>
I am now running grafana-8.5.27 I copied all the folders in the temp structure back to C:\Program Files\GrafanaLabs\grafana.
I tested it from C:\Program Files\GrafanaLabs\grafana\bin>grafana-server.exe which is the output in the prior message. Then I closed that CMD, and re-started the service.
It seems to be running fine with 8.5.27, but I’m unsure why V9 did not migrate also, or what steps to take next.
Maybe try smaller steps: 9.0, 9.1, etc up to 9.5.19.?
I would say that newer Grafana binaries are built with newer Golang 1.21+, which doesn’t support your old Windows 2008r2 anymore (it is 4 years after end of extended support - why you are using it anyway ).
As announced in the Go 1.20 release notes, Go 1.21 requires at least Windows 10 or Windows Server 2016; support for previous versions has been discontinued.
So if you need it, then build own Grafana binaries with Golang 1.20.14.
).