Loki suddenly exit: SIGSEGV: segmentation violation

Grafana Loki
1

A node in Loki’s memberlist cluster suddenly exited without any abnormalities in memory and CPU, while the other three nodes were running normally without exiting.
The log error which exiting node is as follows:

level=error ts=2025-03-20T05:58:54.772558216Z caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Canceled desc = context canceled" addr=<IP>:3110
level=error ts=2025-03-20T05:58:54.772963141Z caller=scheduler_processor.go:175 component=querier org_id=fake traceID=7001c75f6ae01fc2 msg="error notifying scheduler about finished query" err=EOF addr=<IP>:3110
SIGSEGV: segmentation violation
PC=0x419577 m=14 sigcode=1 addr=0x20

goroutine 0 gp=0xc01974c8c0 m=14 mp=0xc0225ab008 [idle]:
runtime.(*mspan).typePointersOfUnchecked(0xc06379a000?, 0xc02b847230?)
        /usr/local/go/src/runtime/mbitmap_allocheaders.go:202 +0x37 fp=0xc026d4de80 sp=0xc026d4de60 pc=0x419577
runtime.scanobject(0xc000095c68?, 0xc000095c68)
        /usr/local/go/src/runtime/mgcmark.go:1446 +0xb5 fp=0xc026d4df10 sp=0xc026d4de80 pc=0x425455
runtime.gcDrain(0xc000095c68, 0x3)
        /usr/local/go/src/runtime/mgcmark.go:1242 +0x1f4 fp=0xc026d4df78 sp=0xc026d4df10 pc=0x424db4
runtime.gcDrainMarkWorkerDedicated(...)
        /usr/local/go/src/runtime/mgcmark.go:1124
runtime.gcBgMarkWorker.func2()
        /usr/local/go/src/runtime/mgc.go:1387 +0xa5 fp=0xc026d4dfc8 sp=0xc026d4df78 pc=0x421425
runtime.systemstack(0x479126)
        /usr/local/go/src/runtime/asm_amd64.s:509 +0x4a fp=0xc026d4dfd8 sp=0xc026d4dfc8 pc=0x4772ea

goroutine 35 gp=0xc0001e6380 m=14 mp=0xc0225ab008 [GC worker (active)]:
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc0000ad750 sp=0xc0000ad740 pc=0x477288
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1370 +0x1f2 fp=0xc0000ad7e0 sp=0xc0000ad750 pc=0x4210b2
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000ad7e8 sp=0xc0000ad7e0 pc=0x479121
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1234 +0x1c

my loki version is 3.2.0

how to solve the problem ?

2

Is that one container still not running?

Segmentation fault once in a while isn’t particularly concerning in my opinion. If you only have this problem with one container you might want to check your Loki metrics and make sure all nodes are load balanced properly.