Server start failure after 8.5.0 upgrade

Hi everyone ,

I just upgraded my grafana server on my respberrypi to the latest 8.5.0 version…
previously i had the 8.4.7

now, grafana won’t start anymore…
i get a big non-understandable (for me) error…

i have 2 raspberry (1x RPI2B and 1x RPI4)
both installation now are broken…

if anyone can help… ; :grinning:

thanks a lot !
Eric.

On the Raspberry Pi 2 i get:

raspberrypi:~ $ grafana-server
runtime: unexpected return pc for github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10 called from 0x9
stack: frame={sp:0x4fa3c34, fp:0x4fa3c5c} stack=[0x4f84000,0x4fa4000)
0x04fa3bb4:  0x00000000  0x04fa3bc0  0x00000000  0x05092460
0x04fa3bc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x04922900  0x00000180  0x00000170
0x04fa3bd4:  0x048000f0  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
0x04fa3be4:  0x76cef2a8  0x0000016c  0x04800100  0x00000000
0x04fa3bf4:  0x02478480  0x04312cd0  0x00000000  0x00000000
0x04fa3c04:  0x00000000  0x00000000  0x0007492c <runtime.sigpanic+0x000001e8>  0x02478480
0x04fa3c14:  0x04312cd0  0x02092c5c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000002c>  0x048000f0  0x022d9601
0x04fa3c24:  0x04922900  0x023bc5d8  0x048000f0  0x02092c5c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000002c>
0x04fa3c34: <0x00000009  0x050924b4  0x04971e40  0x04922900
0x04fa3c44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00054bb8 <runtime.(*consistentHeapStats).acquire+0x00000068>  0x00035d20 <runtime.(*mcache).refill+0x00000154>
0x04fa3c54:  0x048513ac  0x0000001d >0x0000001e  0x00000101
0x04fa3c64:  0x0000001e  0x00000000  0x0487e0bf  0x00000000
0x04fa3c74:  0x00000000  0x00000012  0x00000016  0x0000000b
0x04fa3c84:  0x0486fca0  0x0486fca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
0x04fa3c94:  0x000396bc <runtime.SetFinalizer+0x00000064>  0x0003991c <runtime.SetFinalizer+0x000002c4>  0x00000001  0x00000012
0x04fa3ca4:  0x00000001  0x00000000  0x00000000  0x00000000
0x04fa3cb4:  0x00000000  0x00000000  0x00000000  0x00000000
0x04fa3cc4:  0x04810110  0x00000008  0x00000008  0x0236f380
0x04fa3cd4:  0x0261df30  0x00020138 <armCas64+0x00000000>
fatal error: unknown caller pc

runtime stack:
runtime.throw({0x28134c0, 0x11})
        /usr/local/go/src/runtime/panic.go:1198 +0x5c
runtime.gentraceback(0x5858c, 0x4fa3c0c, 0x0, 0x48000f0, 0x0, 0x0, 0x7fffffff, 0x7e8d34b4, 0x0, 0x0)
        /usr/local/go/src/runtime/traceback.go:274 +0x1540
runtime.addOneOpenDeferFrame.func1()
        /usr/local/go/src/runtime/panic.go:751 +0x8c
runtime.systemstack()
        /usr/local/go/src/runtime/asm_arm.s:318 +0x60

goroutine 1 [running, locked to thread]:
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_arm.s:275 +0x4 fp=0x4fa3b84 sp=0x4fa3b80 pc=0x90abc
runtime.addOneOpenDeferFrame(0x48000f0, 0x5858c, 0x4fa3c0c)
        /usr/local/go/src/runtime/panic.go:750 +0x70 fp=0x4fa3ba0 sp=0x4fa3b84 pc=0x59bf8
panic({0x2478480, 0x4312cd0})
        /usr/local/go/src/runtime/panic.go:998 +0xe8 fp=0x4fa3c0c sp=0x4fa3ba0 pc=0x5aa28
runtime.panicmem()
        /usr/local/go/src/runtime/panic.go:221 +0x4c fp=0x4fa3c18 sp=0x4fa3c0c pc=0x5858c
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:735 +0x1e8 fp=0x4fa3c30 sp=0x4fa3c18 pc=0x7492c
runtime: unexpected return pc for github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10 called from 0x9
stack: frame={sp:0x4fa3c34, fp:0x4fa3c5c} stack=[0x4f84000,0x4fa4000)
0x04fa3bb4:  0x00000000  0x04fa3bc0  0x00000000  0x05092460
0x04fa3bc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x04922900  0x00000180  0x00000170
0x04fa3bd4:  0x048000f0  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
0x04fa3be4:  0x76cef2a8  0x0000016c  0x04800100  0x00000000
0x04fa3bf4:  0x02478480  0x04312cd0  0x00000000  0x00000000
0x04fa3c04:  0x00000000  0x00000000  0x0007492c <runtime.sigpanic+0x000001e8>  0x02478480
0x04fa3c14:  0x04312cd0  0x02092c5c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000002c>  0x048000f0  0x022d9601
0x04fa3c24:  0x04922900  0x023bc5d8  0x048000f0  0x02092c5c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000002c>
0x04fa3c34: <0x00000009  0x050924b4  0x04971e40  0x04922900
0x04fa3c44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00054bb8 <runtime.(*consistentHeapStats).acquire+0x00000068>  0x00035d20 <runtime.(*mcache).refill+0x00000154>
0x04fa3c54:  0x048513ac  0x0000001d >0x0000001e  0x00000101
0x04fa3c64:  0x0000001e  0x00000000  0x0487e0bf  0x00000000
0x04fa3c74:  0x00000000  0x00000012  0x00000016  0x0000000b
0x04fa3c84:  0x0486fca0  0x0486fca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
0x04fa3c94:  0x000396bc <runtime.SetFinalizer+0x00000064>  0x0003991c <runtime.SetFinalizer+0x000002c4>  0x00000001  0x00000012
0x04fa3ca4:  0x00000001  0x00000000  0x00000000  0x00000000
0x04fa3cb4:  0x00000000  0x00000000  0x00000000  0x00000000
0x04fa3cc4:  0x04810110  0x00000008  0x00000008  0x0236f380
0x04fa3cd4:  0x0261df30  0x00020138 <armCas64+0x00000000>
github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10()
        /drone/src/pkg/services/alerting/notifiers/pushover.go:97 +0x2c fp=0x4fa3c5c sp=0x4fa3c34 pc=0x2092c5c

goroutine 114 [select]:
go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).processQueue(0x4b46780)
        /opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:243 +0xf4
go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor.func1(0x4b46780)
        /opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:111 +0x4c
created by go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor
        /opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:109 +0x1e8

goroutine 13 [select]:
go.opencensus.io/stats/view.(*worker).start(0x4b47140)
        /opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xa0
created by go.opencensus.io/stats/view.init.0
        /opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x64

goroutine 146 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x4588760)
        /opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:882 +0x70
created by github.com/golang/glog.init.0
        /opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:410 +0x214

On the Raspberry Pi 4 the error is different:

raspberrypi4:~ $ grafana-server
unexpected fault address 0x27f6b5a
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x27f6b5a pc=0x27f6b5a]

goroutine 1 [running, locked to thread]:
runtime.throw({0x27eb2a9, 0x5})
        /usr/local/go/src/runtime/panic.go:1198 +0x5c fp=0x545bc18 sp=0x545bc04 pc=0x5b244
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:742 +0x1d0 fp=0x545bc30 sp=0x545bc18 pc=0x74914
runtime: unexpected return pc for github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10 called from 0x9
stack: frame={sp:0x545bc34, fp:0x545bc5c} stack=[0x543c000,0x545c000)
0x0545bbb4:  0x0005d1a0 <runtime.gwrite+0x00000034>  0x04668d52  0x029efeac  0x00000001
0x0545bbc4:  0x00000001  0x00000000  0x0005da4c <runtime.printstring+0x00000060>  0x00000002
0x0545bbd4:  0x029efeac  0x00000001  0x00000001  0x00000000
0x0545bbe4:  0x00000000  0x0005b450 <runtime.fatalthrow+0x00000044>  0x0005b244 <runtime.throw+0x0000005c>  0x0545bbf4
0x0545bbf4:  0x0005b468 <runtime.fatalthrow.func1+0x00000000>  0x04c000f0  0x0005b244 <runtime.throw+0x0000005c>  0x0545bc04
0x0545bc04:  0x00074914 <runtime.sigpanic+0x000001d0>  0x0545bc0c  0x0005b25c <runtime.throw.func1+0x00000000>  0x027eb2a9
0x0545bc14:  0x00000005  0x02092c5c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000002c>  0x027eb2a9  0x00000005
0x0545bc24:  0x04e63080  0x027f6b5a  0x04c000f0  0x02092c5c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000002c>
0x0545bc34: <0x00000009  0x05239e18  0x04e03f00  0x04e63080
0x0545bc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00054bb8 <runtime.(*consistentHeapStats).acquire+0x00000068>  0x00035d20 <runtime.(*mcache).refill+0x00000154>
0x0545bc54:  0x04c568ac  0x0000002b >0x0000002c  0x00000101
0x0545bc64:  0x0000002c  0x00000000  0x04caa0bf  0x00000000
0x0545bc74:  0x00000000  0x00000013  0x00000016  0x0000000b
0x0545bc84:  0x04c6eca0  0x04c6eca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
0x0545bc94:  0x000396bc <runtime.SetFinalizer+0x00000064>  0x0003991c <runtime.SetFinalizer+0x000002c4>  0x00000001  0x00000013
0x0545bca4:  0x00000001  0x00000000  0x00000000  0x00000000
0x0545bcb4:  0x00000000  0x00000000  0x00000000  0x00000000
0x0545bcc4:  0x04ca00f8  0x00000008  0x00000008  0x0236f380
0x0545bcd4:  0x0261df30  0x00020138 <armCas64+0x00000000>
github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10()
        /drone/src/pkg/services/alerting/notifiers/pushover.go:97 +0x2c fp=0x545bc5c sp=0x545bc34 pc=0x2092c5c

goroutine 100 [select]:
go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).processQueue(0x4cd49c0)
        /opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:243 +0xf4
go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor.func1(0x4cd49c0)
        /opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:111 +0x4c
created by go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor
        /opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:109 +0x1e8

goroutine 77 [select]:
go.opencensus.io/stats/view.(*worker).start(0x51e8ba0)
        /opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xa0
created by go.opencensus.io/stats/view.init.0
        /opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x64

goroutine 81 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x4588760)
        /opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:882 +0x70
created by github.com/golang/glog.init.0
        /opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:410 +0x214

i just do a rollback of the raspberrypi4 grafana server to 8.4.7 and now it is running again…
so, something is wrong with the latest 8.5.0

i use repo: https://packages.grafana.com/oss/deb stable InRelease

1 Like

Same problem here on Raspi 4… :see_no_evil:

1 Like

same here:

install procedure:

wget https://dl.grafana.com/oss/release/grafana_8.5.0_armhf.deb
sudo dpkg -i grafana_8.5.0_armhf.deb

relevant syslog entries (part1, due to size limitation):

systemd[1]: Started Grafana instance.
kernel: [87125.912581] Unhandled prefetch abort: unknown 1 (0x001) at 0x027f6b5a
grafana-server[30606]: SIGBUS: bus error
kernel: [87126.724895] Unhandled prefetch abort: unknown 1 (0x001) at 0x027f6b5a
grafana-server[30606]: PC=0x27f6b5a m=0 sigcode=0
grafana-server[30606]: goroutine 1 [running, locked to thread]:
grafana-server[30606]: runtime: unknown pc 0x27f6b5a
grafana-server[30606]: stack: frame={sp:0x6c1bc34, fp:0x0} stack=[0x6bfc000,0x6c1c000)
grafana-server[30606]: 0x06c1bbb4:  0x00000000  0x06c1bbd0  0x00000000  0x0651cfc0
grafana-server[30606]: 0x06c1bbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x06652600  0x00000180  0x00000170
grafana-server[30606]: 0x06c1bbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
grafana-server[30606]: 0x06c1bbe4:  0x6596f5a8  0x0000016c  0x00000170  0x76d532e0
grafana-server[30606]: 0x06c1bbf4:  0x00000180  0x045896c8  0x06652600  0x00000000
grafana-server[30606]: 0x06c1bc04:  0x24595180  0x02092c4c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
grafana-server[30606]: 0x06c1bc14:  0x00000001  0x06652600  0x0006d184 <runtime.doInit+0x00000114>  0x022d96a8
grafana-server[30606]: 0x06c1bc24:  0x06652600  0x023bc5d8  0x06601f80  0x027f6b5a
grafana-server[30606]: 0x06c1bc34: <0x00000009  0x0651d020  0x06595140  0x06652600
grafana-server[30606]: 0x06c1bc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00054bb8 <runtime.(*consistentHeapStats).acquire+0x00000068>  0x00035d20 <runtime.(*mcache).refill+0x00000154>
grafana-server[30606]: 0x06c1bc54:  0x06453dac  0x00000017  0x00000018  0x00000101
grafana-server[30606]: 0x06c1bc64:  0x00555555 <google.golang.org/protobuf/internal/impl.newSingularConverter+0x00000e11>  0x065120ff  0x00000018  0x00000000
grafana-server[30606]: 0x06c1bc74:  0x00000000  0x00000018  0x00000016  0x0000000b
grafana-server[30606]: 0x06c1bc84:  0x00000002  0x0646aca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
grafana-server[30606]: 0x06c1bc94:  0x00000008  0x00000019  0x00000001  0x00000019
grafana-server[30606]: 0x06c1bca4:  0x00000001  0x00000000  0x00000000  0x00000000
grafana-server[30606]: runtime: unknown pc 0x27f6b5a
grafana-server[30606]: stack: frame={sp:0x6c1bc34, fp:0x0} stack=[0x6bfc000,0x6c1c000)
grafana-server[30606]: 0x06c1bbb4:  0x00000000  0x06c1bbd0  0x00000000  0x0651cfc0
grafana-server[30606]: 0x06c1bbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x06652600  0x00000180  0x00000170
grafana-server[30606]: 0x06c1bbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
grafana-server[30606]: 0x06c1bbe4:  0x6596f5a8  0x0000016c  0x00000170  0x76d532e0
systemd[1]: grafana-server.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
grafana-server[30606]: 0x06c1bbf4:  0x00000180  0x045896c8  0x06652600  0x00000000
grafana-server[30606]: 0x06c1bc04:  0x24595180  0x02092c4c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
grafana-server[30606]: 0x06c1bc14:  0x00000001  0x06652600  0x0006d184 <runtime.doInit+0x00000114>  0x022d96a8
grafana-server[30606]: 0x06c1bc24:  0x06652600  0x023bc5d8  0x06601f80  0x027f6b5a
grafana-server[30606]: 0x06c1bc34: <0x00000009  0x0651d020  0x06595140  0x06652600
grafana-server[30606]: 0x06c1bc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00054bb8 <runtime.(*consistentHeapStats).acquire+0x00000068>  0x00035d20 <runtime.(*mcache).refill+0x00000154>
grafana-server[30606]: 0x06c1bc54:  0x06453dac  0x00000017  0x00000018  0x00000101
grafana-server[30606]: 0x06c1bc64:  0x00555555 <google.golang.org/protobuf/internal/impl.newSingularConverter+0x00000e11>  0x065120ff  0x00000018  0x00000000
grafana-server[30606]: 0x06c1bc74:  0x00000000  0x00000018  0x00000016  0x0000000b
grafana-server[30606]: 0x06c1bc84:  0x00000002  0x0646aca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
grafana-server[30606]: 0x06c1bc94:  0x00000008  0x00000019  0x00000001  0x00000019
grafana-server[30606]: 0x06c1bca4:  0x00000001  0x00000000  0x00000000  0x00000000
grafana-server[30606]: goroutine 101 [select]:
grafana-server[30606]: go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).processQueue(0x66567e0)
grafana-server[30606]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:243 +0xf4
grafana-server[30606]: go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor.func1(0x66567e0)
grafana-server[30606]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:111 +0x4c
grafana-server[30606]: created by go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor
grafana-server[30606]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:109 +0x1e8
grafana-server[30606]: goroutine 116 [select]:
grafana-server[30606]: go.opencensus.io/stats/view.(*worker).start(0x66d6000)
grafana-server[30606]: #011/opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xa0
grafana-server[30606]: created by go.opencensus.io/stats/view.init.0
grafana-server[30606]: #011/opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x64
grafana-server[30606]: goroutine 96 [chan receive]:
grafana-server[30606]: github.com/golang/glog.(*loggingT).flushDaemon(0x4588760)
systemd[1]: grafana-server.service: Failed with result 'exit-code'.
grafana-server[30606]: #011/opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:882 +0x70
grafana-server[30606]: created by github.com/golang/glog.init.0
grafana-server[30606]: #011/opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:410 +0x214
grafana-server[30606]: trap    0x0
grafana-server[30606]: error   0x1
grafana-server[30606]: oldmask 0x0
grafana-server[30606]: r0      0x6652600
grafana-server[30606]: r1      0x2f1624c
grafana-server[30606]: r2      0x6652600
grafana-server[30606]: r3      0x1
grafana-server[30606]: r4      0x6d184
grafana-server[30606]: r5      0x22d96a8
grafana-server[30606]: r6      0x6652600
grafana-server[30606]: r7      0x23bc5d8
grafana-server[30606]: r8      0x6601f80
grafana-server[30606]: r9      0x180
grafana-server[30606]: r10     0x64000f0
grafana-server[30606]: fp      0x466769c
grafana-server[30606]: ip      0x665277f
grafana-server[30606]: sp      0x6c1bc34
grafana-server[30606]: lr      0x2092c5c
grafana-server[30606]: pc      0x27f6b5a
grafana-server[30606]: cpsr    0x60000010
grafana-server[30606]: fault   0x0
systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 1.
grafana-server[30618]: SIGBUS: bus error
grafana-server[30618]: PC=0x27f6b5a m=0 sigcode=0
grafana-server[30618]: goroutine 1 [running, locked to thread]:
grafana-server[30618]: runtime: unknown pc 0x27f6b5a
grafana-server[30618]: stack: frame={sp:0x6fedc34, fp:0x0} stack=[0x6fce000,0x6fee000)
grafana-server[30618]: 0x06fedbb4:  0x00000000  0x06fedbd0  0x00000000  0x06a82700
grafana-server[30618]: 0x06fedbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x06e40780  0x00000180  0x00000170
grafana-server[30618]: 0x06fedbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
grafana-server[30618]: 0x06fedbe4:  0x646c0e48  0x0000016c  0x00000170  0x76d28790
grafana-server[30618]: 0x06fedbf4:  0x00000180  0x045896c8  0x06e40780  0x00000000
grafana-server[30618]: 0x06fedc04:  0xe092ddc0  0x02092c4c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
grafana-server[30618]: 0x06fedc14:  0x00000001  0x06e40780  0x0006d184 <runtime.doInit+0x00000114>  0x022d96a8
grafana-server[30618]: 0x06fedc24:  0x06e40780  0x023bc5d8  0x06a08040  0x027f6b5a
grafana-server[30618]: 0x06fedc34: <0x00000009  0x06a82758  0x0692dd80  0x06e40780
grafana-server[30618]: 0x06fedc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00000000  0x00000000
grafana-server[30618]: 0x06fedc54:  0x00000000  0x00000000  0x00000000  0x00000000
grafana-server[30618]: 0x06fedc64:  0x00555555 <google.golang.org/protobuf/internal/impl.newSingularConverter+0x00000e11>  0x068a213f  0x00000018  0x00000000
grafana-server[30618]: 0x06fedc74:  0x00000000  0x00000018  0x00000016  0x0000000b
grafana-server[30618]: 0x06fedc84:  0x00000002  0x0686aca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
grafana-server[30618]: 0x06fedc94:  0x00000008  0x0000001a  0x00000001  0x0000001a
grafana-server[30618]: 0x06fedca4:  0x00000001  0x00000000  0x00000000  0x00000000
grafana-server[30618]: runtime: unknown pc 0x27f6b5a
grafana-server[30618]: stack: frame={sp:0x6fedc34, fp:0x0} stack=[0x6fce000,0x6fee000)
grafana-server[30618]: 0x06fedbb4:  0x00000000  0x06fedbd0  0x00000000  0x06a82700
grafana-server[30618]: 0x06fedbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x06e40780  0x00000180  0x00000170
grafana-server[30618]: 0x06fedbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
grafana-server[30618]: 0x06fedbe4:  0x646c0e48  0x0000016c  0x00000170  0x76d28790
grafana-server[30618]: 0x06fedbf4:  0x00000180  0x045896c8  0x06e40780  0x00000000
systemd[1]: Stopped Grafana instance.
grafana-server[30618]: 0x06fedc04:  0xe092ddc0  0x02092c4c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
grafana-server[30618]: 0x06fedc14:  0x00000001  0x06e40780  0x0006d184 <runtime.doInit+0x00000114>  0x022d96a8
grafana-server[30618]: 0x06fedc24:  0x06e40780  0x023bc5d8  0x06a08040  0x027f6b5a
grafana-server[30618]: 0x06fedc34: <0x00000009  0x06a82758  0x0692dd80  0x06e40780
grafana-server[30618]: 0x06fedc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00000000  0x00000000
grafana-server[30618]: 0x06fedc54:  0x00000000  0x00000000  0x00000000  0x00000000
grafana-server[30618]: 0x06fedc64:  0x00555555 <google.golang.org/protobuf/internal/impl.newSingularConverter+0x00000e11>  0x068a213f  0x00000018  0x00000000
grafana-server[30618]: 0x06fedc74:  0x00000000  0x00000018  0x00000016  0x0000000b
grafana-server[30618]: 0x06fedc84:  0x00000002  0x0686aca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
grafana-server[30618]: 0x06fedc94:  0x00000008  0x0000001a  0x00000001  0x0000001a
grafana-server[30618]: 0x06fedca4:  0x00000001  0x00000000  0x00000000  0x00000000
grafana-server[30618]: goroutine 117 [select]:
grafana-server[30618]: go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).processQueue(0x6b086c0)
grafana-server[30618]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:243 +0xf4
grafana-server[30618]: go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor.func1(0x6b086c0)
grafana-server[30618]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:111 +0x4c
grafana-server[30618]: created by go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor
grafana-server[30618]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:109 +0x1e8
grafana-server[30618]: goroutine 12 [select]:
grafana-server[30618]: go.opencensus.io/stats/view.(*worker).start(0x6ad09c0)
grafana-server[30618]: #011/opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xa0
grafana-server[30618]: created by go.opencensus.io/stats/view.init.0
grafana-server[30618]: #011/opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x64
grafana-server[30618]: goroutine 135 [chan receive]:
grafana-server[30618]: github.com/golang/glog.(*loggingT).flushDaemon(0x4588760)
grafana-server[30618]: #011/opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:882 +0x70
grafana-server[30618]: created by github.com/golang/glog.init.0
systemd[1]: Started Grafana instance.
grafana-server[30618]: #011/opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:410 +0x214
grafana-server[30618]: trap    0x0
grafana-server[30618]: error   0x1
grafana-server[30618]: oldmask 0x0
grafana-server[30618]: r0      0x6e40780
grafana-server[30618]: r1      0x2f1624c
grafana-server[30618]: r2      0x6e40780
grafana-server[30618]: r3      0x1
grafana-server[30618]: r4      0x6d184
grafana-server[30618]: r5      0x22d96a8
grafana-server[30618]: r6      0x6e40780
grafana-server[30618]: r7      0x23bc5d8
grafana-server[30618]: r8      0x6a08040
grafana-server[30618]: r9      0x180
grafana-server[30618]: r10     0x68000f0
grafana-server[30618]: fp      0x466769c
grafana-server[30618]: ip      0x6e408ff
grafana-server[30618]: sp      0x6fedc34
grafana-server[30618]: lr      0x2092c5c
grafana-server[30618]: pc      0x27f6b5a
grafana-server[30618]: cpsr    0x60000010
grafana-server[30618]: fault   0x0
systemd[1]: grafana-server.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
systemd[1]: grafana-server.service: Failed with result 'exit-code'.
systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 2.
systemd[1]: Stopped Grafana instance.
systemd[1]: Started Grafana instance.
kernel: [87127.654566] Unhandled prefetch abort: unknown 1 (0x001) at 0x027f6b5a
grafana-server[30639]: SIGBUS: bus error
grafana-server[30639]: PC=0x27f6b5a m=0 sigcode=0
grafana-server[30639]: goroutine 1 [running, locked to thread]:
grafana-server[30639]: runtime: unknown pc 0x27f6b5a
grafana-server[30639]: stack: frame={sp:0x586dc34, fp:0x0} stack=[0x584e000,0x586e000)
grafana-server[30639]: 0x0586dbb4:  0x00000000  0x0586dbc0  0x00000000  0x050bb180
grafana-server[30639]: 0x0586dbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x0548e900  0x00000180  0x00000170
grafana-server[30639]: 0x0586dbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
grafana-server[30639]: 0x0586dbe4:  0x64fc8620  0x0000016c  0x00000170  0x76d7f538
grafana-server[30639]: 0x0586dbf4:  0x00000180  0x045896c8  0x0548e900  0x00000000
grafana-server[30639]: 0x0586dc04:  0xc6042500  0x02092c4c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
grafana-server[30639]: 0x0586dc14:  0x00000001  0x0548e900  0x0006d184 <runtime.doInit+0x00000114>  0x022d96a8
grafana-server[30639]: 0x0586dc24:  0x0548e900  0x023bc5d8  0x050a0b80  0x027f6b5a
grafana-server[30639]: 0x0586dc34: <0x00000009  0x050bb1d8  0x05042300  0x0548e900
grafana-server[30639]: 0x0586dc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00000000  0x00000000
grafana-server[30639]: 0x0586dc54:  0x00000000  0x00000000  0x00000000  0x00000000
grafana-server[30639]: 0x0586dc64:  0x00555555 <google.golang.org/protobuf/internal/impl.newSingularConverter+0x00000e11>  0x050a213f  0x00000018  0x00000000
grafana-server[30639]: 0x0586dc74:  0x00000000  0x00000018  0x00000016  0x0000000b
grafana-server[30639]: 0x0586dc84:  0x00000002  0x0506aca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
grafana-server[30639]: 0x0586dc94:  0x00000008  0x00000014  0x00000001  0x00000014
grafana-server[30639]: 0x0586dca4:  0x00000001  0x00000000  0x00000000  0x00000000
grafana-server[30639]: runtime: unknown pc 0x27f6b5a
grafana-server[30639]: stack: frame={sp:0x586dc34, fp:0x0} stack=[0x584e000,0x586e000)
grafana-server[30639]: 0x0586dbb4:  0x00000000  0x0586dbc0  0x00000000  0x050bb180
grafana-server[30639]: 0x0586dbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x0548e900  0x00000180  0x00000170
grafana-server[30639]: 0x0586dbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
grafana-server[30639]: 0x0586dbe4:  0x64fc8620  0x0000016c  0x00000170  0x76d7f538
grafana-server[30639]: 0x0586dbf4:  0x00000180  0x045896c8  0x0548e900  0x00000000
systemd[1]: grafana-server.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
grafana-server[30639]: 0x0586dc04:  0xc6042500  0x02092c4c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
grafana-server[30639]: 0x0586dc14:  0x00000001  0x0548e900  0x0006d184 <runtime.doInit+0x00000114>  0x022d96a8
grafana-server[30639]: 0x0586dc24:  0x0548e900  0x023bc5d8  0x050a0b80  0x027f6b5a
grafana-server[30639]: 0x0586dc34: <0x00000009  0x050bb1d8  0x05042300  0x0548e900
grafana-server[30639]: 0x0586dc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00000000  0x00000000
grafana-server[30639]: 0x0586dc54:  0x00000000  0x00000000  0x00000000  0x00000000
grafana-server[30639]: 0x0586dc64:  0x00555555 <google.golang.org/protobuf/internal/impl.newSingularConverter+0x00000e11>  0x050a213f  0x00000018  0x00000000
grafana-server[30639]: 0x0586dc74:  0x00000000  0x00000018  0x00000016  0x0000000b
grafana-server[30639]: 0x0586dc84:  0x00000002  0x0506aca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
grafana-server[30639]: 0x0586dc94:  0x00000008  0x00000014  0x00000001  0x00000014
grafana-server[30639]: 0x0586dca4:  0x00000001  0x00000000  0x00000000  0x00000000
grafana-server[30639]: goroutine 89 [select]:

relevant syslog entries (part2, due to size limitation):

grafana-server[30639]: go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).processQueue(0x522f140)
grafana-server[30639]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:243 +0xf4
grafana-server[30639]: go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor.func1(0x522f140)
grafana-server[30639]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:111 +0x4c
grafana-server[30639]: created by go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor
grafana-server[30639]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:109 +0x1e8
grafana-server[30639]: goroutine 80 [select]:
grafana-server[30639]: go.opencensus.io/stats/view.(*worker).start(0x522f8c0)
grafana-server[30639]: #011/opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xa0
grafana-server[30639]: created by go.opencensus.io/stats/view.init.0
grafana-server[30639]: #011/opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x64
grafana-server[30639]: goroutine 134 [chan receive]:
grafana-server[30639]: github.com/golang/glog.(*loggingT).flushDaemon(0x4588760)
grafana-server[30639]: #011/opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:882 +0x70
grafana-server[30639]: created by github.com/golang/glog.init.0
systemd[1]: grafana-server.service: Failed with result 'exit-code'.
grafana-server[30639]: #011/opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:410 +0x214
grafana-server[30639]: trap    0x0
grafana-server[30639]: error   0x1
grafana-server[30639]: oldmask 0x0
grafana-server[30639]: r0      0x548e900
grafana-server[30639]: r1      0x2f1624c
grafana-server[30639]: r2      0x548e900
grafana-server[30639]: r3      0x1
grafana-server[30639]: r4      0x6d184
grafana-server[30639]: r5      0x22d96a8
grafana-server[30639]: r6      0x548e900
grafana-server[30639]: r7      0x23bc5d8
grafana-server[30639]: r8      0x50a0b80
grafana-server[30639]: r9      0x180
grafana-server[30639]: r10     0x50000f0
grafana-server[30639]: fp      0x466769c
grafana-server[30639]: ip      0x548ea7f
grafana-server[30639]: sp      0x586dc34
grafana-server[30639]: lr      0x2092c5c
grafana-server[30639]: pc      0x27f6b5a
grafana-server[30639]: cpsr    0x60000010
grafana-server[30639]: fault   0x0
systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 3.
systemd[1]: Stopped Grafana instance.
systemd[1]: Started Grafana instance.
kernel: [87128.443909] Unhandled prefetch abort: unknown 1 (0x001) at 0x027f6b5a
grafana-server[30669]: SIGBUS: bus error
grafana-server[30669]: PC=0x27f6b5a m=0 sigcode=0
grafana-server[30669]: goroutine 1 [running, locked to thread]:
grafana-server[30669]: runtime: unknown pc 0x27f6b5a
grafana-server[30669]: stack: frame={sp:0x6c6dc34, fp:0x0} stack=[0x6c4e000,0x6c6e000)
grafana-server[30669]: 0x06c6dbb4:  0x00000000  0x06c6dbc0  0x00000000  0x066ece00
grafana-server[30669]: 0x06c6dbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x06724d80  0x00000180  0x00000170
grafana-server[30669]: 0x06c6dbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
grafana-server[30669]: 0x06c6dbe4:  0x76d7e768  0x0000016c  0x00000170  0x76d74538
grafana-server[30669]: 0x06c6dbf4:  0x00000180  0x045896c8  0x06724d80  0x00000000
grafana-server[30669]: 0x06c6dc04:  0x71616200  0x02092c4c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
grafana-server[30669]: 0x06c6dc14:  0x00000001  0x06724d80  0x0006d184 <runtime.doInit+0x00000114>  0x022d96a8
grafana-server[30669]: 0x06c6dc24:  0x06724d80  0x023bc5d8  0x065993c0  0x027f6b5a
grafana-server[30669]: 0x06c6dc34: <0x00000009  0x066ece60  0x06616180  0x06724d80
grafana-server[30669]: 0x06c6dc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00000000  0x00000000
grafana-server[30669]: 0x06c6dc54:  0x00000000  0x00000000  0x00000000  0x00000000
grafana-server[30669]: 0x06c6dc64:  0x00555555 <google.golang.org/protobuf/internal/impl.newSingularConverter+0x00000e11>  0x064a213f  0x00000018  0x00000000
grafana-server[30669]: 0x06c6dc74:  0x00000000  0x00000018  0x00000016  0x0000000b
grafana-server[30669]: 0x06c6dc84:  0x00000002  0x0646aca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
grafana-server[30669]: 0x06c6dc94:  0x00000008  0x00000019  0x00000001  0x00000019
grafana-server[30669]: 0x06c6dca4:  0x00000001  0x00000000  0x00000000  0x00000000
grafana-server[30669]: runtime: unknown pc 0x27f6b5a
grafana-server[30669]: stack: frame={sp:0x6c6dc34, fp:0x0} stack=[0x6c4e000,0x6c6e000)
grafana-server[30669]: 0x06c6dbb4:  0x00000000  0x06c6dbc0  0x00000000  0x066ece00
grafana-server[30669]: 0x06c6dbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x06724d80  0x00000180  0x00000170
grafana-server[30669]: 0x06c6dbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
grafana-server[30669]: 0x06c6dbe4:  0x76d7e768  0x0000016c  0x00000170  0x76d74538
grafana-server[30669]: 0x06c6dbf4:  0x00000180  0x045896c8  0x06724d80  0x00000000
systemd[1]: grafana-server.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
grafana-server[30669]: 0x06c6dc04:  0x71616200  0x02092c4c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
grafana-server[30669]: 0x06c6dc14:  0x00000001  0x06724d80  0x0006d184 <runtime.doInit+0x00000114>  0x022d96a8
grafana-server[30669]: 0x06c6dc24:  0x06724d80  0x023bc5d8  0x065993c0  0x027f6b5a
grafana-server[30669]: 0x06c6dc34: <0x00000009  0x066ece60  0x06616180  0x06724d80
grafana-server[30669]: 0x06c6dc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00000000  0x00000000
grafana-server[30669]: 0x06c6dc54:  0x00000000  0x00000000  0x00000000  0x00000000
grafana-server[30669]: 0x06c6dc64:  0x00555555 <google.golang.org/protobuf/internal/impl.newSingularConverter+0x00000e11>  0x064a213f  0x00000018  0x00000000
grafana-server[30669]: 0x06c6dc74:  0x00000000  0x00000018  0x00000016  0x0000000b
grafana-server[30669]: 0x06c6dc84:  0x00000002  0x0646aca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
grafana-server[30669]: 0x06c6dc94:  0x00000008  0x00000019  0x00000001  0x00000019
grafana-server[30669]: 0x06c6dca4:  0x00000001  0x00000000  0x00000000  0x00000000
grafana-server[30669]: goroutine 95 [select]:
grafana-server[30669]: go.opencensus.io/stats/view.(*worker).start(0x6721260)
grafana-server[30669]: #011/opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xa0
grafana-server[30669]: created by go.opencensus.io/stats/view.init.0
grafana-server[30669]: #011/opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x64
grafana-server[30669]: goroutine 119 [select]:
grafana-server[30669]: go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).processQueue(0x6720360)
grafana-server[30669]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:243 +0xf4
grafana-server[30669]: go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor.func1(0x6720360)
grafana-server[30669]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:111 +0x4c
grafana-server[30669]: created by go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor
grafana-server[30669]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:109 +0x1e8
grafana-server[30669]: goroutine 125 [chan receive]:
grafana-server[30669]: github.com/golang/glog.(*loggingT).flushDaemon(0x4588760)
grafana-server[30669]: #011/opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:882 +0x70
grafana-server[30669]: created by github.com/golang/glog.init.0
systemd[1]: grafana-server.service: Failed with result 'exit-code'.
grafana-server[30669]: #011/opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:410 +0x214
grafana-server[30669]: trap    0x0
grafana-server[30669]: error   0x1
grafana-server[30669]: oldmask 0x0
grafana-server[30669]: r0      0x6724d80
grafana-server[30669]: r1      0x2f1624c
grafana-server[30669]: r2      0x6724d80
grafana-server[30669]: r3      0x1
grafana-server[30669]: r4      0x6d184
grafana-server[30669]: r5      0x22d96a8
grafana-server[30669]: r6      0x6724d80
grafana-server[30669]: r7      0x23bc5d8
grafana-server[30669]: r8      0x65993c0
grafana-server[30669]: r9      0x180
grafana-server[30669]: r10     0x64000f0
grafana-server[30669]: fp      0x466769c
grafana-server[30669]: ip      0x6724eff
grafana-server[30669]: sp      0x6c6dc34
grafana-server[30669]: lr      0x2092c5c
grafana-server[30669]: pc      0x27f6b5a
grafana-server[30669]: cpsr    0x60000010
grafana-server[30669]: fault   0x0
systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 4.
systemd[1]: Stopped Grafana instance.
systemd[1]: Started Grafana instance.
kernel: [87129.191852] Unhandled prefetch abort: unknown 1 (0x001) at 0x027f6b5a
grafana-server[30689]: SIGBUS: bus error
grafana-server[30689]: PC=0x27f6b5a m=0 sigcode=0
grafana-server[30689]: goroutine 1 [running, locked to thread]:
grafana-server[30689]: runtime: unknown pc 0x27f6b5a
grafana-server[30689]: stack: frame={sp:0x6c6dc34, fp:0x0} stack=[0x6c4e000,0x6c6e000)
grafana-server[30689]: 0x06c6dbb4:  0x00000000  0x06c6dbc0  0x00000000  0x0671fc00
grafana-server[30689]: 0x06c6dbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x06743c80  0x00000180  0x00000170
grafana-server[30689]: 0x06c6dbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
grafana-server[30689]: 0x06c6dbe4:  0x76d5ab68  0x0000016c  0x00000170  0x76d4f088
grafana-server[30689]: 0x06c6dbf4:  0x00000180  0x045896c8  0x06743c80  0x00000000
grafana-server[30689]: 0x06c6dc04:  0x0962fc00  0x02092c4c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
grafana-server[30689]: 0x06c6dc14:  0x00000001  0x06743c80  0x0006d184 <runtime.doInit+0x00000114>  0x022d96a8
grafana-server[30689]: 0x06c6dc24:  0x06743c80  0x023bc5d8  0x0659d240  0x027f6b5a
grafana-server[30689]: 0x06c6dc34: <0x00000009  0x0671fc58  0x0662fb80  0x06743c80
grafana-server[30689]: 0x06c6dc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00000000  0x00000000
grafana-server[30689]: 0x06c6dc54:  0x00000000  0x00000000  0x00000000  0x00000000
grafana-server[30689]: 0x06c6dc64:  0x00555555 <google.golang.org/protobuf/internal/impl.newSingularConverter+0x00000e11>  0x0644213f  0x00000018  0x00000000
grafana-server[30689]: 0x06c6dc74:  0x00000000  0x00000018  0x00000016  0x0000000b
grafana-server[30689]: 0x06c6dc84:  0x00000002  0x0646fca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
grafana-server[30689]: 0x06c6dc94:  0x00000008  0x00000019  0x00000001  0x00000019
grafana-server[30689]: 0x06c6dca4:  0x00000001  0x00000000  0x00000000  0x00000000
grafana-server[30689]: runtime: unknown pc 0x27f6b5a
grafana-server[30689]: stack: frame={sp:0x6c6dc34, fp:0x0} stack=[0x6c4e000,0x6c6e000)
grafana-server[30689]: 0x06c6dbb4:  0x00000000  0x06c6dbc0  0x00000000  0x0671fc00
grafana-server[30689]: 0x06c6dbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x06743c80  0x00000180  0x00000170
grafana-server[30689]: 0x06c6dbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sendWindowUpdate32+0x000000f0>  0x027f0100
grafana-server[30689]: 0x06c6dbe4:  0x76d5ab68  0x0000016c  0x00000170  0x76d4f088
grafana-server[30689]: 0x06c6dbf4:  0x00000180  0x045896c8  0x06743c80  0x00000000
systemd[1]: grafana-server.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
grafana-server[30689]: 0x06c6dc04:  0x0962fc00  0x02092c4c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
grafana-server[30689]: 0x06c6dc14:  0x00000001  0x06743c80  0x0006d184 <runtime.doInit+0x00000114>  0x022d96a8
grafana-server[30689]: 0x06c6dc24:  0x06743c80  0x023bc5d8  0x0659d240  0x027f6b5a
grafana-server[30689]: 0x06c6dc34: <0x00000009  0x0671fc58  0x0662fb80  0x06743c80
grafana-server[30689]: 0x06c6dc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00000000  0x00000000
grafana-server[30689]: 0x06c6dc54:  0x00000000  0x00000000  0x00000000  0x00000000
grafana-server[30689]: 0x06c6dc64:  0x00555555 <google.golang.org/protobuf/internal/impl.newSingularConverter+0x00000e11>  0x0644213f  0x00000018  0x00000000
grafana-server[30689]: 0x06c6dc74:  0x00000000  0x00000018  0x00000016  0x0000000b
grafana-server[30689]: 0x06c6dc84:  0x00000002  0x0646fca4  0x0000001a  0x00033400 <runtime.findObject+0x00000058>
grafana-server[30689]: 0x06c6dc94:  0x00000008  0x00000019  0x00000001  0x00000019
grafana-server[30689]: 0x06c6dca4:  0x00000001  0x00000000  0x00000000  0x00000000
grafana-server[30689]: goroutine 105 [select]:
grafana-server[30689]: go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).processQueue(0x66350e0)
grafana-server[30689]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:243 +0xf4
grafana-server[30689]: go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor.func1(0x66350e0)
grafana-server[30689]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:111 +0x4c
grafana-server[30689]: created by go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor
grafana-server[30689]: #011/opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_span_processor.go:109 +0x1e8
grafana-server[30689]: goroutine 111 [select]:
grafana-server[30689]: go.opencensus.io/stats/view.(*worker).start(0x66354a0)
grafana-server[30689]: #011/opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xa0
grafana-server[30689]: created by go.opencensus.io/stats/view.init.0
grafana-server[30689]: #011/opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x64
grafana-server[30689]: goroutine 135 [chan receive]:
grafana-server[30689]: github.com/golang/glog.(*loggingT).flushDaemon(0x4588760)
grafana-server[30689]: #011/opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:882 +0x70
grafana-server[30689]: created by github.com/golang/glog.init.0
systemd[1]: grafana-server.service: Failed with result 'exit-code'.
grafana-server[30689]: #011/opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:410 +0x214
grafana-server[30689]: trap    0x0
grafana-server[30689]: error   0x1
grafana-server[30689]: oldmask 0x0
grafana-server[30689]: r0      0x6743c80
grafana-server[30689]: r1      0x2f1624c
grafana-server[30689]: r2      0x6743c80
grafana-server[30689]: r3      0x1
grafana-server[30689]: r4      0x6d184
grafana-server[30689]: r5      0x22d96a8
grafana-server[30689]: r6      0x6743c80
grafana-server[30689]: r7      0x23bc5d8
grafana-server[30689]: r8      0x659d240
grafana-server[30689]: r9      0x180
grafana-server[30689]: r10     0x64000f0
grafana-server[30689]: fp      0x466769c
grafana-server[30689]: ip      0x6743dff
grafana-server[30689]: sp      0x6c6dc34
grafana-server[30689]: lr      0x2092c5c
grafana-server[30689]: pc      0x27f6b5a
grafana-server[30689]: cpsr    0x60000010
grafana-server[30689]: fault   0x0
systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 5.
systemd[1]: Stopped Grafana instance.
systemd[1]: grafana-server.service: Start request repeated too quickly.
systemd[1]: grafana-server.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Grafana instance.

I have the same issue. Can I ask how did you do the rollback to the version 8.4.7?

i simply use:

sudo apt install grafana=8.4.7
8 Likes

This is doing the trick. Thank you.

same here, grafana-server 8.5.0 does not run on my Pi 4.
Rollback to 8.4.7 and all is good

pi@raspberrypidns:~ $ grafana-server
unexpected fault address 0x0
fatal error: fault
[signal SIGBUS: bus error code=0x1 addr=0x0 pc=0x27f6b5a]

goroutine 1 [running, locked to thread]:
runtime.throw({0x27eb2a9, 0x5})
/usr/local/go/src/runtime/panic.go:1198 +0x5c fp=0x5d2dc18 sp=0x5d2dc04 pc=0x5b244
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:732 +0xac fp=0x5d2dc30 sp=0x5d2dc18 pc=0x747f0
runtime: unexpected return pc for github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10 called from 0x9
stack: frame={sp:0x5d2dc34, fp:0x5d2dc5c} stack=[0x5d1e000,0x5d2e000)

Exactly same issue on 8.5.0, had to install old version again :frowning:

Same here with a Raspberry PI 3B

SIGBUS: bus error
PC=0x27f6b5a m=0 sigcode=0

goroutine 1 [running, locked to thread]:
runtime: unknown pc 0x27f6b5a
stack: frame={sp:0x6c8fc34, fp:0x0} stack=[0x6c80000,0x6c90000)
0x06c8fbb4:  0x00000000  0x06c8fbc0  0x00000000  0x06a84700
0x06c8fbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x06630900  0x000001800x00000170
0x06c8fbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sdWindowUpdate32+0x000000f0>  0x027f0100
0x06c8fbe4:  0x76d87378  0x0000016c  0x00000170  0x76d7a088
0x06c8fbf4:  0x00000180  0x045896c8  0x06630900  0x00000000
0x06c8fc04:  0x0c69bac0  0x02092c4c <github.com/grafana/grafana/pkg/services/arting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
0x06c8fc14:  0x00000001  0x06630900  0x0006d184 <runtime.doInit+0x00000114>  022d96a8
0x06c8fc24:  0x06630900  0x023bc5d8  0x068c0f00  0x027f6b5a
0x06c8fc34: <0x00000009  0x06a847bc  0x0669ba80  0x06630900
0x06c8fc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00054bb8 <rtime.(*consistentHeapStats).acquire+0x00000068>  0x00035d20 <runtime.(*mcache)efill+0x00000154>
0x06c8fc54:  0x064513ac  0x0000001d  0x0000001e  0x00000101
0x06c8fc64:  0x0000001e  0x00000000  0x0647e0bf  0x00000000
0x06c8fc74:  0x00000000  0x00000019  0x00000016  0x0000000b
0x06c8fc84:  0x0646fca0  0x0646fca4  0x0000001a  0x00033400 <runtime.findObjec0x00000058>
0x06c8fc94:  0x000396bc <runtime.SetFinalizer+0x00000064>  0x0003991c <runtimeetFinalizer+0x000002c4>  0x00000001  0x00000019
0x06c8fca4:  0x00000001  0x00000000  0x00000000  0x00000000
runtime: unknown pc 0x27f6b5a
stack: frame={sp:0x6c8fc34, fp:0x0} stack=[0x6c80000,0x6c90000)
0x06c8fbb4:  0x00000000  0x06c8fbc0  0x00000000  0x06a84700
0x06c8fbc4:  0x0002c194 <runtime.newobject+0x0000002c>  0x06630900  0x000001800x00000170
0x06c8fbd4:  0x022d96a8  0x00000040  0x002c2038 <net/http.(*http2serverConn).sdWindowUpdate32+0x000000f0>  0x027f0100
0x06c8fbe4:  0x76d87378  0x0000016c  0x00000170  0x76d7a088
0x06c8fbf4:  0x00000180  0x045896c8  0x06630900  0x00000000
0x06c8fc04:  0x0c69bac0  0x02092c4c <github.com/grafana/grafana/pkg/services/arting/notifiers.init.10+0x0000001c>  0x00000170  0x022d96a8
0x06c8fc14:  0x00000001  0x06630900  0x0006d184 <runtime.doInit+0x00000114>  022d96a8
0x06c8fc24:  0x06630900  0x023bc5d8  0x068c0f00  0x027f6b5a
0x06c8fc34: <0x00000009  0x06a847bc  0x0669ba80  0x06630900
0x06c8fc44:  0x0006d0c4 <runtime.doInit+0x00000054>  0x0433c6b8  0x00054bb8 <rtime.(*consistentHeapStats).acquire+0x00000068>  0x00035d20 <runtime.(*mcache)efill+0x00000154>
0x06c8fc54:  0x064513ac  0x0000001d  0x0000001e  0x00000101
0x06c8fc64:  0x0000001e  0x00000000  0x0647e0bf  0x00000000
0x06c8fc74:  0x00000000  0x00000019  0x00000016  0x0000000b
0x06c8fc84:  0x0646fca0  0x0646fca4  0x0000001a  0x00033400 <runtime.findObjec0x00000058>
0x06c8fc94:  0x000396bc <runtime.SetFinalizer+0x00000064>  0x0003991c <runtimeetFinalizer+0x000002c4>  0x00000001  0x00000019
0x06c8fca4:  0x00000001  0x00000000  0x00000000  0x00000000

goroutine 81 [select]:
go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).processQueue(0x65249c
        /opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_an_processor.go:243 +0xf4
go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor.func1(0x65249c0)
        /opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_an_processor.go:111 +0x4c
created by go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor
        /opt/drone/gomodcache/go.opentelemetry.io/otel/sdk@v1.0.0/trace/batch_an_processor.go:109 +0x1e8

goroutine 115 [select]:
go.opencensus.io/stats/view.(*worker).start(0x673df20)
        /opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:27+0xa0
created by go.opencensus.io/stats/view.init.0
        /opt/drone/gomodcache/go.opencensus.io@v0.23.0/stats/view/worker.go:340x64

goroutine 122 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x4588760)
        /opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23d4e6c14b/glog.go:882 +0x70
created by github.com/golang/glog.init.0
        /opt/drone/gomodcache/github.com/golang/glog@v0.0.0-20160126235308-23d4e6c14b/glog.go:410 +0x214


1 Like

me too.

/var/log/grafana $ grafana-server
unexpected fault address 0x27f6b5a
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x27f6b5a pc=0x27f6b5a]

goroutine 1 [running, locked to thread]:
runtime.throw({0x27eb2a9, 0x5})
/usr/local/go/src/runtime/panic.go:1198 +0x5c fp=0x549bc18 sp=0x549bc04 pc=0x5b244
runtime.sigpanic()

…etc…

After rolling back to a previous version you might want to put the package “on hold” to prevent that during the next update it is replaced with the current version (which does not run):

sudo apt-mark hold grafana
1 Like

Same here on Raspi 3B+, Pi OS Bullseye:
$ grafana-server
SIGBUS: bus error
PC=0x27f6b5a m=0 sigcode=0

Same here. Raspberry4 running buster. Downgrade fixed the issue.

The above also is reported as a bug: grafana-server doesn't start after recent update to 8.5.0 on openhabian · Issue #48130 · grafana/grafana · GitHub

I experienced the same problem. Currently there is no fix. Although it seems to work on the 64 bit version of Raspberry Pi OS (formerly known as Raspbian).

thanks man its work after 5hour

There is an open issue in github, our developers are actively working on it.

2 Likes

I had the same problem. I’ve downgraded to 8.4.7 and everything works ok!

Same error here:

-- Logs begin at Thu 2022-04-28 09:35:36 BST, end at Thu 2022-04-28 09:52:38 BST. --
Apr 28 09:35:50 boba systemd[1]: Started Grafana instance.
Apr 28 09:35:59 boba grafana-server[542]: unexpected fault address 0x27f6b5a
Apr 28 09:35:59 boba grafana-server[542]: fatal error: fault
Apr 28 09:35:59 boba grafana-server[542]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x27f6b5a pc=0x27f6b5a]
Apr 28 09:35:59 boba grafana-server[542]: goroutine 1 [running, locked to thread]:
Apr 28 09:35:59 boba grafana-server[542]: runtime.throw({0x27eb2a9, 0x5})
Apr 28 09:35:59 boba grafana-server[542]:         /usr/local/go/src/runtime/panic.go:1198 +0x5c fp=0x5f9bc18 sp=0x5f9bc04 pc=0x5b244
Apr 28 09:35:59 boba grafana-server[542]: runtime.sigpanic()
Apr 28 09:35:59 boba grafana-server[542]:         /usr/local/go/src/runtime/signal_unix.go:742 +0x1d0 fp=0x5f9bc30 sp=0x5f9bc18 pc=0x74914
Apr 28 09:35:59 boba grafana-server[542]: runtime: unexpected return pc for github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10 called from 0x9
Apr 28 09:35:59 boba grafana-server[542]: stack: frame={sp:0x5f9bc34, fp:0x5f9bc5c} stack=[0x5f7c000,0x5f9c000)
Apr 28 09:35:59 boba grafana-server[542]: 0x05f9bbb4:  0x0005d1a0 <runtime.gwrite+0x00000034>  0x04668d52  0x029efeac  0x00000001
Apr 28 09:35:59 boba grafana-server[542]: 0x05f9bbc4:  0x00000001  0x00000000  0x0005da4c <runtime.printstring+0x00000060>  0x00000002
Apr 28 09:35:59 boba grafana-server[542]: 0x05f9bbd4:  0x029efeac  0x00000001  0x00000001  0x00000000
Apr 28 09:35:59 boba grafana-server[542]: 0x05f9bbe4:  0x00000000  0x0005b450 <runtime.fatalthrow+0x00000044>  0x0005b244 <runtime.throw+0x0000005c>  0x05f9bbf4
Apr 28 09:35:59 boba grafana-server[542]: 0x05f9bbf4:  0x0005b468 <runtime.fatalthrow.func1+0x00000000>  0x058000f0  0x0005b244 <runtime.throw+0x0000005c>  0x05f9bc04
Apr 28 09:35:59 boba grafana-server[542]: 0x05f9bc04:  0x00074914 <runtime.sigpanic+0x000001d0>  0x05f9bc0c  0x0005b25c <runtime.throw.func1+0x00000000>  0x027eb2a9
Apr 28 09:35:59 boba grafana-server[542]: 0x05f9bc14:  0x00000005  0x02092c5c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000002c>  0x027eb2a9  0x00000005
Apr 28 09:35:59 boba grafana-server[542]: 0x05f9bc24:  0x059eaf00  0x027f6b5a  0x058000f0  0x02092c5c <github.com/grafana/grafana/pkg/services/alerting/notifiers.init.10+0x0000002c>
Apr 28 09:35:59 boba grafana-server[542]: 0x05f9bc34: <0x00000009  0x05dfa3dc  0x05cb2040  0x059eaf00

Raspbian Linux 10 - ARMv7 Processor rev 3 (v7l) @ 270 bMips, 4 cores