Hope everyone is doing great. I had a query and I would really appreciate if someone could try to help in that.
I have one QA environment and one Prod environment(Both Kubernetes). I used helm to deploy grafana,loki stack using the chart “grafana/loki-stack”. I made a slight change in the values.yaml file and that change is that I only changed grafana enabled to “true” and image to “latest”. Here:
grafana:
enabled: true
sidecar:
datasources:
label: “”
labelValue: “”
enabled: true
maxLines: 1000
image:
tag: latest
Now when I deployed it on QA, it is working perfectly fine but when I deploy the same config on Prod its failing. I am attaching Screenshots and logs. Help is much appreciated fellas.
these are the logs:
level=info ts=2025-05-30T13:01:56.622951049Z caller=main.go:103 msg=“Starting Loki” version=“(version=2.6.1, branch=HEAD, revision=6bd05c9a4)”
level=info ts=2025-05-30T13:01:56.623037653Z caller=modules.go:736 msg=“RulerStorage is not configured in single binary mode and will not be started.”
level=info ts=2025-05-30T13:01:56.62355372Z caller=server.go:288 http=[::]:3100 grpc=[::]:9095 msg=“server listening on addresses”
level=warn ts=2025-05-30T13:01:56.624230231Z caller=experimental.go:20 msg=“experimental feature in use” feature=“In-memory (FIFO) cache - chunksfifocache”
level=info ts=2025-05-30T13:01:56.624596455Z caller=table_manager.go:252 msg=“query readiness setup completed” duration=1.092µs distinct_users_len=0
level=info ts=2025-05-30T13:01:56.625890075Z caller=shipper.go:124 msg=“starting index shipper in RW mode”
level=info ts=2025-05-30T13:01:56.626069275Z caller=shipper_index_client.go:79 msg=“starting boltdb shipper in RW mode”
level=info ts=2025-05-30T13:01:56.627832214Z caller=table_manager.go:167 msg=“handing over indexes to shipper”
level=info ts=2025-05-30T13:01:56.630263709Z caller=table_manager.go:134 msg=“uploading tables”
level=info ts=2025-05-30T13:01:56.632637035Z caller=modules.go:761 msg=“RulerStorage is nil. Not starting the ruler.”
level=info ts=2025-05-30T13:01:56.633865901Z caller=worker.go:112 msg=“Starting querier worker using query-scheduler and scheduler ring for addresses”
level=info ts=2025-05-30T13:01:56.636741799Z caller=module_service.go:82 msg=initialising module=server
level=info ts=2025-05-30T13:01:56.636806923Z caller=module_service.go:82 msg=initialising module=memberlist-kv
level=info ts=2025-05-30T13:01:56.63685315Z caller=module_service.go:82 msg=initialising module=query-frontend-tripperware
level=error ts=2025-05-30T13:01:56.636993064Z caller=loki.go:386 msg=“module failed” module=memberlist-kv error=“invalid service state: Failed, expected: Running, failure: service &{0xc0007845a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc000140a00 loki false { false}} 0xc000140a00 loki [{ringDesc 0xe43cc0} {}]} 0xc000306c80 0xc000140a00 0xc0003efea0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc000350240 0xc000b0c360 0xc000b0c420 0xc000b0c4e0 0xc000b0c5a0 0xc000b0c7e0 0xc000b0c660 0xc000b0c8a0 0xc000b0c720 0xc000ae2500 0xc000ae2540 0xc000b0ca20 0xc000b0cae0 0xc000b0cc60 0xc000b0cba0 0xc000134190 0xc0002d1d50 0xc000134180 0xc000134188 0xc000ae2580 0xc000ae25c0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”
level=error ts=2025-05-30T13:01:56.637062026Z caller=loki.go:386 msg=“module failed” module=query-scheduler error=“failed to start query-scheduler, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0007845a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc000140a00 loki false { false}} 0xc000140a00 loki [{ringDesc 0xe43cc0} {}]} 0xc000306c80 0xc000140a00 0xc0003efea0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc000350240 0xc000b0c360 0xc000b0c420 0xc000b0c4e0 0xc000b0c5a0 0xc000b0c7e0 0xc000b0c660 0xc000b0c8a0 0xc000b0c720 0xc000ae2500 0xc000ae2540 0xc000b0ca20 0xc000b0cae0 0xc000b0cc60 0xc000b0cba0 0xc000134190 0xc0002d1d50 0xc000134180 0xc000134188 0xc000ae2580 0xc000ae25c0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”
level=error ts=2025-05-30T13:01:56.637103273Z caller=loki.go:386 msg=“module failed” module=ingester-querier error=“failed to start ingester-querier, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0007845a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc000140a00 loki false { false}} 0xc000140a00 loki [{ringDesc 0xe43cc0} {}]} 0xc000306c80 0xc000140a00 0xc0003efea0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc000350240 0xc000b0c360 0xc000b0c420 0xc000b0c4e0 0xc000b0c5a0 0xc000b0c7e0 0xc000b0c660 0xc000b0c8a0 0xc000b0c720 0xc000ae2500 0xc000ae2540 0xc000b0ca20 0xc000b0cae0 0xc000b0cc60 0xc000b0cba0 0xc000134190 0xc0002d1d50 0xc000134180 0xc000134188 0xc000ae2580 0xc000ae25c0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”
level=error ts=2025-05-30T13:01:56.637125555Z caller=loki.go:386 msg=“module failed” module=distributor error=“failed to start distributor, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0007845a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc000140a00 loki false { false}} 0xc000140a00 loki [{ringDesc 0xe43cc0} {}]} 0xc000306c80 0xc000140a00 0xc0003efea0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc000350240 0xc000b0c360 0xc000b0c420 0xc000b0c4e0 0xc000b0c5a0 0xc000b0c7e0 0xc000b0c660 0xc000b0c8a0 0xc000b0c720 0xc000ae2500 0xc000ae2540 0xc000b0ca20 0xc000b0cae0 0xc000b0cc60 0xc000b0cba0 0xc000134190 0xc0002d1d50 0xc000134180 0xc000134188 0xc000ae2580 0xc000ae25c0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”
level=error ts=2025-05-30T13:01:56.637155161Z caller=loki.go:386 msg=“module failed” module=ring error=“failed to start ring, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0007845a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc000140a00 loki false { false}} 0xc000140a00 loki [{ringDesc 0xe43cc0} {}]} 0xc000306c80 0xc000140a00 0xc0003efea0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc000350240 0xc000b0c360 0xc000b0c420 0xc000b0c4e0 0xc000b0c5a0 0xc000b0c7e0 0xc000b0c660 0xc000b0c8a0 0xc000b0c720 0xc000ae2500 0xc000ae2540 0xc000b0ca20 0xc000b0cae0 0xc000b0cc60 0xc000b0cba0 0xc000134190 0xc0002d1d50 0xc000134180 0xc000134188 0xc000ae2580 0xc000ae25c0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”
level=info ts=2025-05-30T13:01:56.63719697Z caller=module_service.go:114 msg=“module stopped” module=query-frontend-tripperware
level=error ts=2025-05-30T13:01:56.637187243Z caller=loki.go:386 msg=“module failed” module=usage-report error=“failed to start usage-report, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0007845a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc000140a00 loki false { false}} 0xc000140a00 loki [{ringDesc 0xe43cc0} {}]} 0xc000306c80 0xc000140a00 0xc0003efea0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc000350240 0xc000b0c360 0xc000b0c420 0xc000b0c4e0 0xc000b0c5a0 0xc000b0c7e0 0xc000b0c660 0xc000b0c8a0 0xc000b0c720 0xc000ae2500 0xc000ae2540 0xc000b0ca20 0xc000b0cae0 0xc000b0cc60 0xc000b0cba0 0xc000134190 0xc0002d1d50 0xc000134180 0xc000134188 0xc000ae2580 0xc000ae25c0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”
level=error ts=2025-05-30T13:01:56.637293204Z caller=loki.go:386 msg=“module failed” module=compactor error=“failed to start compactor, because it depends on module ring, which has failed: context canceled”
level=error ts=2025-05-30T13:01:56.637304324Z caller=loki.go:386 msg=“module failed” module=store error=“failed to start store, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0007845a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc000140a00 loki false { false}} 0xc000140a00 loki [{ringDesc 0xe43cc0} {}]} 0xc000306c80 0xc000140a00 0xc0003efea0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc000350240 0xc000b0c360 0xc000b0c420 0xc000b0c4e0 0xc000b0c5a0 0xc000b0c7e0 0xc000b0c660 0xc000b0c8a0 0xc000b0c720 0xc000ae2500 0xc000ae2540 0xc000b0ca20 0xc000b0cae0 0xc000b0cc60 0xc000b0cba0 0xc000134190 0xc0002d1d50 0xc000134180 0xc000134188 0xc000ae2580 0xc000ae25c0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”
level=error ts=2025-05-30T13:01:56.637314413Z caller=loki.go:386 msg=“module failed” module=ingester error=“failed to start ingester, because it depends on module store, which has failed: context canceled”
level=error ts=2025-05-30T13:01:56.637322438Z caller=loki.go:386 msg=“module failed” module=querier error=“failed to start querier, because it depends on module store, which has failed: context canceled”
level=error ts=2025-05-30T13:01:56.637348929Z caller=loki.go:386 msg=“module failed” module=query-frontend error=“failed to start query-frontend, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0007845a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc000140a00 loki false { false}} 0xc000140a00 loki [{ringDesc 0xe43cc0} {}]} 0xc000306c80 0xc000140a00 0xc0003efea0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc000350240 0xc000b0c360 0xc000b0c420 0xc000b0c4e0 0xc000b0c5a0 0xc000b0c7e0 0xc000b0c660 0xc000b0c8a0 0xc000b0c720 0xc000ae2500 0xc000ae2540 0xc000b0ca20 0xc000b0cae0 0xc000b0cc60 0xc000b0cba0 0xc000134190 0xc0002d1d50 0xc000134180 0xc000134188 0xc000ae2580 0xc000ae25c0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”
level=info ts=2025-05-30T13:01:56.638533522Z caller=modules.go:1090 msg=“server stopped”
level=info ts=2025-05-30T13:01:56.63855342Z caller=module_service.go:114 msg=“module stopped” module=server
level=info ts=2025-05-30T13:01:56.638567627Z caller=loki.go:375 msg=“Loki stopped”
failed services
github.com/grafana/loki/pkg/loki.(*Loki).Run
/src/loki/pkg/loki/loki.go:421
main.main
/src/loki/cmd/loki/main.go:105
runtime.main
/usr/local/go/src/runtime/proc.go:255
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1581
level=error ts=2025-05-30T13:01:56.638634383Z caller=log.go:103 msg=“error running loki” err=“failed services\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:421\nmain.main\n\t/src/loki/cmd/loki/main.go:105\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581”