Issue while using grafana/loki-stack helm chart

Hope everyone is doing great. I had a query and I would really appreciate if someone could try to help in that.
I have one QA environment and one Prod environment(Both Kubernetes). I used helm to deploy grafana,loki stack using the chart “grafana/loki-stack”. I made a slight change in the values.yaml file and that change is that I only changed grafana enabled to “true” and image to “latest”. Here:
grafana:
enabled: true
sidecar:
datasources:
label: “”
labelValue: “”
enabled: true
maxLines: 1000
image:
tag: latest
Now when I deployed it on QA, it is working perfectly fine but when I deploy the same config on Prod its failing. I am attaching Screenshots and logs. Help is much appreciated fellas.

these are the logs:

level=info ts=2025-05-30T13:01:56.622951049Z caller=main.go:103 msg=“Starting Loki” version=“(version=2.6.1, branch=HEAD, revision=6bd05c9a4)”
level=info ts=2025-05-30T13:01:56.623037653Z caller=modules.go:736 msg=“RulerStorage is not configured in single binary mode and will not be started.”
level=info ts=2025-05-30T13:01:56.62355372Z caller=server.go:288 http=[::]:3100 grpc=[::]:9095 msg=“server listening on addresses”
level=warn ts=2025-05-30T13:01:56.624230231Z caller=experimental.go:20 msg=“experimental feature in use” feature=“In-memory (FIFO) cache - chunksfifocache”
level=info ts=2025-05-30T13:01:56.624596455Z caller=table_manager.go:252 msg=“query readiness setup completed” duration=1.092µs distinct_users_len=0
level=info ts=2025-05-30T13:01:56.625890075Z caller=shipper.go:124 msg=“starting index shipper in RW mode”
level=info ts=2025-05-30T13:01:56.626069275Z caller=shipper_index_client.go:79 msg=“starting boltdb shipper in RW mode”
level=info ts=2025-05-30T13:01:56.627832214Z caller=table_manager.go:167 msg=“handing over indexes to shipper”
level=info ts=2025-05-30T13:01:56.630263709Z caller=table_manager.go:134 msg=“uploading tables”
level=info ts=2025-05-30T13:01:56.632637035Z caller=modules.go:761 msg=“RulerStorage is nil. Not starting the ruler.”
level=info ts=2025-05-30T13:01:56.633865901Z caller=worker.go:112 msg=“Starting querier worker using query-scheduler and scheduler ring for addresses”
level=info ts=2025-05-30T13:01:56.636741799Z caller=module_service.go:82 msg=initialising module=server
level=info ts=2025-05-30T13:01:56.636806923Z caller=module_service.go:82 msg=initialising module=memberlist-kv
level=info ts=2025-05-30T13:01:56.63685315Z caller=module_service.go:82 msg=initialising module=query-frontend-tripperware
level=error ts=2025-05-30T13:01:56.636993064Z caller=loki.go:386 msg=“module failed” module=memberlist-kv error=“invalid service state: Failed, expected: Running, failure: service &{0xc0007845a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc000140a00 loki false { false}} 0xc000140a00 loki [{ringDesc 0xe43cc0} {}]} 0xc000306c80 0xc000140a00 0xc0003efea0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc000350240 0xc000b0c360 0xc000b0c420 0xc000b0c4e0 0xc000b0c5a0 0xc000b0c7e0 0xc000b0c660 0xc000b0c8a0 0xc000b0c720 0xc000ae2500 0xc000ae2540 0xc000b0ca20 0xc000b0cae0 0xc000b0cc60 0xc000b0cba0 0xc000134190 0xc0002d1d50 0xc000134180 0xc000134188 0xc000ae2580 0xc000ae25c0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”
level=error ts=2025-05-30T13:01:56.637062026Z caller=loki.go:386 msg=“module failed” module=query-scheduler error=“failed to start query-scheduler, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0007845a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc000140a00 loki false { false}} 0xc000140a00 loki [{ringDesc 0xe43cc0} {}]} 0xc000306c80 0xc000140a00 0xc0003efea0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc000350240 0xc000b0c360 0xc000b0c420 0xc000b0c4e0 0xc000b0c5a0 0xc000b0c7e0 0xc000b0c660 0xc000b0c8a0 0xc000b0c720 0xc000ae2500 0xc000ae2540 0xc000b0ca20 0xc000b0cae0 0xc000b0cc60 0xc000b0cba0 0xc000134190 0xc0002d1d50 0xc000134180 0xc000134188 0xc000ae2580 0xc000ae25c0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”
level=error ts=2025-05-30T13:01:56.637103273Z caller=loki.go:386 msg=“module failed” module=ingester-querier error=“failed to start ingester-querier, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0007845a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc000140a00 loki false { false}} 0xc000140a00 loki [{ringDesc 0xe43cc0} {}]} 0xc000306c80 0xc000140a00 0xc0003efea0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc000350240 0xc000b0c360 0xc000b0c420 0xc000b0c4e0 0xc000b0c5a0 0xc000b0c7e0 0xc000b0c660 0xc000b0c8a0 0xc000b0c720 0xc000ae2500 0xc000ae2540 0xc000b0ca20 0xc000b0cae0 0xc000b0cc60 0xc000b0cba0 0xc000134190 0xc0002d1d50 0xc000134180 0xc000134188 0xc000ae2580 0xc000ae25c0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”
level=error ts=2025-05-30T13:01:56.637125555Z caller=loki.go:386 msg=“module failed” module=distributor error=“failed to start distributor, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0007845a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc000140a00 loki false { false}} 0xc000140a00 loki [{ringDesc 0xe43cc0} {}]} 0xc000306c80 0xc000140a00 0xc0003efea0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc000350240 0xc000b0c360 0xc000b0c420 0xc000b0c4e0 0xc000b0c5a0 0xc000b0c7e0 0xc000b0c660 0xc000b0c8a0 0xc000b0c720 0xc000ae2500 0xc000ae2540 0xc000b0ca20 0xc000b0cae0 0xc000b0cc60 0xc000b0cba0 0xc000134190 0xc0002d1d50 0xc000134180 0xc000134188 0xc000ae2580 0xc000ae25c0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”
level=error ts=2025-05-30T13:01:56.637155161Z caller=loki.go:386 msg=“module failed” module=ring error=“failed to start ring, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0007845a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc000140a00 loki false { false}} 0xc000140a00 loki [{ringDesc 0xe43cc0} {}]} 0xc000306c80 0xc000140a00 0xc0003efea0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc000350240 0xc000b0c360 0xc000b0c420 0xc000b0c4e0 0xc000b0c5a0 0xc000b0c7e0 0xc000b0c660 0xc000b0c8a0 0xc000b0c720 0xc000ae2500 0xc000ae2540 0xc000b0ca20 0xc000b0cae0 0xc000b0cc60 0xc000b0cba0 0xc000134190 0xc0002d1d50 0xc000134180 0xc000134188 0xc000ae2580 0xc000ae25c0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”
level=info ts=2025-05-30T13:01:56.63719697Z caller=module_service.go:114 msg=“module stopped” module=query-frontend-tripperware
level=error ts=2025-05-30T13:01:56.637187243Z caller=loki.go:386 msg=“module failed” module=usage-report error=“failed to start usage-report, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0007845a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc000140a00 loki false { false}} 0xc000140a00 loki [{ringDesc 0xe43cc0} {}]} 0xc000306c80 0xc000140a00 0xc0003efea0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc000350240 0xc000b0c360 0xc000b0c420 0xc000b0c4e0 0xc000b0c5a0 0xc000b0c7e0 0xc000b0c660 0xc000b0c8a0 0xc000b0c720 0xc000ae2500 0xc000ae2540 0xc000b0ca20 0xc000b0cae0 0xc000b0cc60 0xc000b0cba0 0xc000134190 0xc0002d1d50 0xc000134180 0xc000134188 0xc000ae2580 0xc000ae25c0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”
level=error ts=2025-05-30T13:01:56.637293204Z caller=loki.go:386 msg=“module failed” module=compactor error=“failed to start compactor, because it depends on module ring, which has failed: context canceled”
level=error ts=2025-05-30T13:01:56.637304324Z caller=loki.go:386 msg=“module failed” module=store error=“failed to start store, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0007845a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc000140a00 loki false { false}} 0xc000140a00 loki [{ringDesc 0xe43cc0} {}]} 0xc000306c80 0xc000140a00 0xc0003efea0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc000350240 0xc000b0c360 0xc000b0c420 0xc000b0c4e0 0xc000b0c5a0 0xc000b0c7e0 0xc000b0c660 0xc000b0c8a0 0xc000b0c720 0xc000ae2500 0xc000ae2540 0xc000b0ca20 0xc000b0cae0 0xc000b0cc60 0xc000b0cba0 0xc000134190 0xc0002d1d50 0xc000134180 0xc000134188 0xc000ae2580 0xc000ae25c0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”
level=error ts=2025-05-30T13:01:56.637314413Z caller=loki.go:386 msg=“module failed” module=ingester error=“failed to start ingester, because it depends on module store, which has failed: context canceled”
level=error ts=2025-05-30T13:01:56.637322438Z caller=loki.go:386 msg=“module failed” module=querier error=“failed to start querier, because it depends on module store, which has failed: context canceled”
level=error ts=2025-05-30T13:01:56.637348929Z caller=loki.go:386 msg=“module failed” module=query-frontend error=“failed to start query-frontend, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0007845a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc000140a00 loki false { false}} 0xc000140a00 loki [{ringDesc 0xe43cc0} {}]} 0xc000306c80 0xc000140a00 0xc0003efea0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc000350240 0xc000b0c360 0xc000b0c420 0xc000b0c4e0 0xc000b0c5a0 0xc000b0c7e0 0xc000b0c660 0xc000b0c8a0 0xc000b0c720 0xc000ae2500 0xc000ae2540 0xc000b0ca20 0xc000b0cae0 0xc000b0cc60 0xc000b0cba0 0xc000134190 0xc0002d1d50 0xc000134180 0xc000134188 0xc000ae2580 0xc000ae25c0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”
level=info ts=2025-05-30T13:01:56.638533522Z caller=modules.go:1090 msg=“server stopped”
level=info ts=2025-05-30T13:01:56.63855342Z caller=module_service.go:114 msg=“module stopped” module=server
level=info ts=2025-05-30T13:01:56.638567627Z caller=loki.go:375 msg=“Loki stopped”
failed services
github.com/grafana/loki/pkg/loki.(*Loki).Run
/src/loki/pkg/loki/loki.go:421
main.main
/src/loki/cmd/loki/main.go:105
runtime.main
/usr/local/go/src/runtime/proc.go:255
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1581
level=error ts=2025-05-30T13:01:56.638634383Z caller=log.go:103 msg=“error running loki” err=“failed services\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:421\nmain.main\n\t/src/loki/cmd/loki/main.go:105\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581”

This sounds like the problem. I’d recommend you to double check your kubernetes cluster.

What to check exactly?? I have 3 nodes cluster on QA and same also another 3 nodes cluster for Production. It worked perfectly fine on QA but giving this on prod

HI All, am getting same issue when tryin to install using helm /loki-stack

evel=error ts=2025-06-02T12:58:32.926191423Z caller=loki.go:386 msg=“module failed” module=memberlist-kv error=“invalid service state: Failed, expected: Running, failure: service &{0xc00094d900 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc0000a4a50 loki false { false}} 0xc0000a4a50 loki [{ringDesc 0xe43cc0} {}]} 0xc0008eb7c0 0xc0000a4a50 0xc0002d61c0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc0007fd860 0xc000235620 0xc0002356e0 0xc0002357a0 0xc000235860 0xc000235aa0 0xc000235920 0xc000235b60 0xc0002359e0 0xc00036cf40 0xc00036d0c0 0xc000235ce0 0xc000235da0 0xc000235f20 0xc000235e60 0xc0005384b8 0xc0002d6bd0 0xc0005384a8 0xc0005384b0 0xc00036d200 0xc00036d280 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”

what should we check on the cluster? pod to pod comm is fine, pinged and its fine

Please share your configuration.

loki:
enabled: true
isDefault: true
url: http://{{ (include “loki.serviceName” .) }}:{{ .Values.loki.service.port }}
readinessProbe:
httpGet:
path: /ready
port: http-metrics
initialDelaySeconds: 45
livenessProbe:
httpGet:
path: /ready
port: http-metrics
initialDelaySeconds: 45
extraArgs:
config.expand-env: “true”
extraEnvVars:
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
datasource:
jsonData: “{}”
uid: “”

This is my values .yaml:

test_pod:
enabled: true
image: bats/bats:1.8.2
pullPolicy: IfNotPresent

loki:
storage:
memberlist:
join_members:
- “loki-0.loki-headless:3100” # First node in the cluster
- “loki-1.loki-headless:3100” # Second node in the cluster
- “loki-2.loki-headless:3100” # Third node in the cluster
ring:
kvstore:
store: inmemory
auth_enabled: false
commonConfig:
replication_factor: 3 # Set the replication factor to 3 for 3 nodes
enabled: true
isDefault: true
url: http://{{(include “loki.serviceName” .)}}:{{ .Values.loki.service.port }}
readinessProbe:
httpGet:
path: /ready
port: http-metrics
initialDelaySeconds: 45
livenessProbe:
httpGet:
path: /ready
port: http-metrics
initialDelaySeconds: 45
datasource:
jsonData: “{}”
uid: “”

promtail:
enabled: true
config:
logLevel: info
serverPort: 3101
clients:
- url: http://{{ .Release.Name }}:3100/loki/api/v1/push

fluent-bit:
enabled: false

grafana:
enabled: true
sidecar:
datasources:
label: “”
labelValue: “”
enabled: true
maxLines: 1000
image:
tag: latest

prometheus:
enabled: false
isDefault: false
url: http://{{ include “prometheus.fullname” .}}:{{ .Values.prometheus.server.service.servicePort }}{{ .Values.prometheus.server.prefixURL }}
datasource:
jsonData: “{}”

filebeat:
enabled: false
filebeatConfig:
filebeat.yml: |
# logging.level: debug
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: “/var/log/containers/”
output.logstash:
hosts: [“logstash-loki:5044”]

logstash:
enabled: false
image: grafana/logstash-output-loki
imageTag: 1.0.1
filters:
main: |-
filter {
if [kubernetes] {
mutate {
add_field => {
“container_name” => “%{[kubernetes][container][name]}”
“namespace” => “%{[kubernetes][namespace]}”
“pod” => “%{[kubernetes][pod][name]}”
}
replace => { “host” => “%{[kubernetes][node][name]}”}
}
}
mutate {
remove_field => [“tags”]
}
}
outputs:
main: |-
output {
loki {
url => “http://loki:3100/loki/api/v1/push
#username => “test”
#password => “test”
}
# stdout { codec => rubydebug }
}

proxy:
http_proxy: “”
https_proxy: “”
no_proxy: “”

Can you check your pod’s env variable (and maybe the configmap as well) and see if the IP passed in is actually correct?

@tonyswumac Can you check mine as well??