After upgrading to 9.3.2 Grafana enterprise crashes in my raspberry pi with a nil pointer exception.
No information is added to log files. It looks like it cashes early in the start up process.
I reverted to version 9.3.1. It works again.
So something introduced in 9.3.2.
You have a paid version?
No, I am using the free version to visualize some home automation data.
the above is a paid version. But that might not be the issue.
Look into the grafana log file for specific errors.
There is no information stored in the grafana log files. The application seems to crash before any log lines are written. When I look at at the console output via journalctl (grafana-server runs as systemctl service). I get the following output:
Dec 19 08:37:28 rpimeter systemd[1]: Started Grafana instance.
Dec 19 08:37:29 rpimeter grafana-server[28548]: panic: runtime error: invalid memory address or nil pointer dereference
Dec 19 08:37:29 rpimeter grafana-server[28548]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x28db49c]
Dec 19 08:37:29 rpimeter grafana-server[28548]: goroutine 1 [running]:
Dec 19 08:37:29 rpimeter grafana-server[28548]: github.com/grafana/grafana/pkg/extensions/config/vault.init()
Dec 19 08:37:29 rpimeter grafana-server[28548]: /drone/src/pkg/extensions/config/vault/instrumentation.go:21 +0x15c
Dec 19 08:37:29 rpimeter systemd[1]: grafana-server.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Dec 19 08:37:29 rpimeter systemd[1]: grafana-server.service: Failed with result 'exit-code'.
Dec 19 08:37:30 rpimeter systemd[1]: grafana-server.service: Service RestartSec=100ms expired, scheduling restart.
Dec 19 08:37:30 rpimeter systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 11.
Dec 19 08:37:30 rpimeter systemd[1]: Stopped Grafana instance.
Dec 19 08:37:30 rpimeter systemd[1]: grafana-server.service: Start request repeated too quickly.
Dec 19 08:37:30 rpimeter systemd[1]: grafana-server.service: Failed with result 'exit-code'.
Dec 19 08:37:30 rpimeter systemd[1]: Failed to start Grafana instance.
1 Like
What if you install 9.3.2 separately on a dofferent port and see if it works
I upgraded to grafana 9.3.2 again and changed the port number from 3000 to 4444. No difference observed. Still crashes.
I am using this source to update/install/downgrade grafana via apt:
deb https://packages.grafana.com/enterprise/deb stable main
I am running OS version:
Distributor ID: Raspbian
Description: Raspbian GNU/Linux 10 (buster)
Release: 10
Codename: buster
And kernel version:
Linux rpimeter 5.10.103-v7+ #1529 SMP Tue Mar 8 12:21:37 GMT 2022 armv7l GNU/Linux
What I recommended was not an upgrade but a separate parallel clean install to eliminate the upgrade as a possible source of the issue
Ok, I misinterpreted. Now I did the following:
As root (sudo su -):
- Create a folder /root/grafana932
- Change owner to grafana:grafana
- Change to /root/grafana932
- Download: wget https://dl.grafana.com/enterprise/release/grafana-enterprise-9.3.2.linux-armv7.tar.gz
- Unpack: tar -zxvf grafana-enterprise-9.3.2.linux-armv7.tar.gz
- Change owner recursive to grafana:grafana
- Copy config file /etc/grafana/grafana.ini to /root/grafana/grafana-9.3.2/conf
- Change owner of copied config file to grafana:grafana
- Change port number in config file to 4444
- Stop ‘original’ grafana to prevent conflicts in log files, etc.
- Run: sudo -u grafana -g grafana bin/grafana-server --config=/root/grafana/grafana-9.3.2/conf/grafana.ini
This is the output:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x28db49c]
goroutine 1 [running]:
github.com/grafana/grafana/pkg/extensions/config/vault.init()
/drone/src/pkg/extensions/config/vault/instrumentation.go:21 +0x15c
This is the exact same error as before.
1 Like
Good, so now we know that it is not an upgrade issue but that specific version has a problem.
Might be best to post this as a bug in grafana github
To check if this does work with the previous version I repeated the above for 9.3.1 i.s.o. 9.3.2. This gave the following results:
Failed to parse defaults.ini, open /root/grafana/grafana-9.3.1/conf/defaults.ini: permission denied
I created /home/grafana and moved the grafana folder from /root to /home/grafana and repeated the startups:
For 9.3.2:
root@rpimeter:/home/grafana/grafana/grafana-9.3.2# sudo -u grafana -g grafana bin/grafana-server --config=/home/grafana/grafana/grafana-9.3.2/conf/grafana.ini
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x28db49c]
goroutine 1 [running]:
github.com/grafana/grafana/pkg/extensions/config/vault.init()
/drone/src/pkg/extensions/config/vault/instrumentation.go:21 +0x15c
For 9.3.1:
root@rpimeter:/home/grafana/grafana/grafana-9.3.1# sudo -u grafana -g grafana bin/grafana-server --config=/home/grafana/grafana/grafana-9.3.1/conf/grafana.ini
WARN [12-19|17:41:24] Skipping finding plugins as directory does not exist logger=plugin.finder path=/home/grafana/grafana/grafana-9.3.1/data/plugins
WARN [12-19|17:41:26] secret scan interval is too low, increasing to 5m0s logger=serviceaccounts.background
WARN [12-19|17:41:26] Scheduling and sending of reports disabled, no image renderer found/installed. For image rendering support please install the grafana-image-renderer plugin. Read more at https://grafana.com/docs/grafana/latest/administration/image_rendering/ logger=report
The grafana server (9.3.1) is running.
It looks like a change in 9.3.2 from 9.3.1 introduced a null pointer exception / causes a segmentation fault.
1 Like
Someone else with the same problem created an issue on github. I added info and link to this discussion.
See: Grafana panics during start after update from 9.3.1 to 9.3.2 on raspberry Pi · Issue #60463 · grafana/grafana · GitHub
1 Like