Not able to install grafana loki on eks

while installing grafana loki on eks one of my pods is failing

pod name - loki-0

error message-

level=info ts=2023-08-28T08:02:09.961731067Z caller=main.go:103 msg=“Starting Loki” version=“(version=2.6.1, branch=HEAD, revision=6bd05c9a4)”

level=info ts=2023-08-28T08:02:09.962036523Z caller=server.go:288 http=[::]:3100 grpc=[::]:9095 msg=“server listening on addresses”

level=info ts=2023-08-28T08:02:09.962398655Z caller=modules.go:736 msg=“RulerStorage is not configured in single binary mode and will not be started.”

level=warn ts=2023-08-28T08:02:09.964184608Z caller=experimental.go:20 msg=“experimental feature in use” feature=“In-memory (FIFO) cache - chunksfifocache”

level=info ts=2023-08-28T08:02:09.964580407Z caller=table_manager.go:252 msg=“query readiness setup completed” duration=3.548µs distinct_users_len=0

level=info ts=2023-08-28T08:02:09.964606039Z caller=shipper.go:124 msg=“starting index shipper in RW mode”

level=info ts=2023-08-28T08:02:09.964693442Z caller=shipper_index_client.go:79 msg=“starting boltdb shipper in RW mode”

level=info ts=2023-08-28T08:02:09.965141586Z caller=worker.go:112 msg=“Starting querier worker using query-scheduler and scheduler ring for addresses”

level=info ts=2023-08-28T08:02:09.96590728Z caller=modules.go:761 msg=“RulerStorage is nil. Not starting the ruler.”

level=info ts=2023-08-28T08:02:09.967635418Z caller=table_manager.go:134 msg=“uploading tables”

level=info ts=2023-08-28T08:02:09.967796294Z caller=table_manager.go:167 msg=“handing over indexes to shipper”

level=info ts=2023-08-28T08:02:09.970299462Z caller=module_service.go:82 msg=initialising module=server

level=info ts=2023-08-28T08:02:09.970433871Z caller=module_service.go:82 msg=initialising module=query-frontend-tripperware

level=info ts=2023-08-28T08:02:09.970471177Z caller=module_service.go:82 msg=initialising module=memberlist-kv

level=error ts=2023-08-28T08:02:09.970553126Z caller=loki.go:386 msg=“module failed” module=memberlist-kv error=“invalid service state: Failed, expected: Running, failure: service &{0xc0001e7f40 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc000094a50 loki false { false}} 0xc000094a50 loki [{ringDesc 0xe43cc0} {}]} 0xc00012b720 0xc000094a50 0xc0007688c0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc00023a540 0xc0006e13e0 0xc0006e14a0 0xc0006e1560 0xc0006e1620 0xc0006e1860 0xc0006e16e0 0xc0006e1920 0xc0006e17a0 0xc00060e8c0 0xc00060e900 0xc0006e1aa0 0xc0006e1b60 0xc0006e1ce0 0xc0006e1c20 0xc00000e5b0 0xc000769500 0xc00000e5a0 0xc00000e5a8 0xc00060e940 0xc00060e980 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”

level=error ts=2023-08-28T08:02:09.970600683Z caller=loki.go:386 msg=“module failed” module=query-frontend error=“failed to start query-frontend, because it depends on module query-scheduler, which has failed: context canceled”

level=error ts=2023-08-28T08:02:09.970639914Z caller=loki.go:386 msg=“module failed” module=distributor error=“failed to start distributor, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0001e7f40 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc000094a50 loki false { false}} 0xc000094a50 loki [{ringDesc 0xe43cc0} {}]} 0xc00012b720 0xc000094a50 0xc0007688c0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc00023a540 0xc0006e13e0 0xc0006e14a0 0xc0006e1560 0xc0006e1620 0xc0006e1860 0xc0006e16e0 0xc0006e1920 0xc0006e17a0 0xc00060e8c0 0xc00060e900 0xc0006e1aa0 0xc0006e1b60 0xc0006e1ce0 0xc0006e1c20 0xc00000e5b0 0xc000769500 0xc00000e5a0 0xc00000e5a8 0xc00060e940 0xc00060e980 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”

level=error ts=2023-08-28T08:02:09.970665799Z caller=loki.go:386 msg=“module failed” module=store error=“failed to start store, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0001e7f40 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc000094a50 loki false { false}} 0xc000094a50 loki [{ringDesc 0xe43cc0} {}]} 0xc00012b720 0xc000094a50 0xc0007688c0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc00023a540 0xc0006e13e0 0xc0006e14a0 0xc0006e1560 0xc0006e1620 0xc0006e1860 0xc0006e16e0 0xc0006e1920 0xc0006e17a0 0xc00060e8c0 0xc00060e900 0xc0006e1aa0 0xc0006e1b60 0xc0006e1ce0 0xc0006e1c20 0xc00000e5b0 0xc000769500 0xc00000e5a0 0xc00000e5a8 0xc00060e940 0xc00060e980 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”

level=error ts=2023-08-28T08:02:09.970687529Z caller=loki.go:386 msg=“module failed” module=ingester error=“failed to start ingester, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0001e7f40 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc000094a50 loki false { false}} 0xc000094a50 loki [{ringDesc 0xe43cc0} {}]} 0xc00012b720 0xc000094a50 0xc0007688c0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc00023a540 0xc0006e13e0 0xc0006e14a0 0xc0006e1560 0xc0006e1620 0xc0006e1860 0xc0006e16e0 0xc0006e1920 0xc0006e17a0 0xc00060e8c0 0xc00060e900 0xc0006e1aa0 0xc0006e1b60 0xc0006e1ce0 0xc0006e1c20 0xc00000e5b0 0xc000769500 0xc00000e5a0 0xc00000e5a8 0xc00060e940 0xc00060e980 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”

level=info ts=2023-08-28T08:02:09.970722135Z caller=module_service.go:114 msg=“module stopped” module=query-frontend-tripperware

level=error ts=2023-08-28T08:02:09.970741589Z caller=loki.go:386 msg=“module failed” module=query-scheduler error=“failed to start query-scheduler, because it depends on module usage-report, which has failed: context canceled”

level=error ts=2023-08-28T08:02:09.970773199Z caller=loki.go:386 msg=“module failed” module=compactor error=“failed to start compactor, because it depends on module usage-report, which has failed: context canceled”

level=error ts=2023-08-28T08:02:09.970837482Z caller=loki.go:386 msg=“module failed” module=ring error=“failed to start ring, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0001e7f40 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc000094a50 loki false { false}} 0xc000094a50 loki [{ringDesc 0xe43cc0} {}]} 0xc00012b720 0xc000094a50 0xc0007688c0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc00023a540 0xc0006e13e0 0xc0006e14a0 0xc0006e1560 0xc0006e1620 0xc0006e1860 0xc0006e16e0 0xc0006e1920 0xc0006e17a0 0xc00060e8c0 0xc00060e900 0xc0006e1aa0 0xc0006e1b60 0xc0006e1ce0 0xc0006e1c20 0xc00000e5b0 0xc000769500 0xc00000e5a0 0xc00000e5a8 0xc00060e940 0xc00060e980 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”

level=error ts=2023-08-28T08:02:09.97087122Z caller=loki.go:386 msg=“module failed” module=ingester-querier error=“failed to start ingester-querier, because it depends on module ring, which has failed: context canceled”

level=error ts=2023-08-28T08:02:09.970985738Z caller=loki.go:386 msg=“module failed” module=querier error=“failed to start querier, because it depends on module ingester-querier, which has failed: context canceled”

level=error ts=2023-08-28T08:02:09.971182069Z caller=loki.go:386 msg=“module failed” module=usage-report error=“failed to start usage-report, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0001e7f40 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true 7946 false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 { 7946 5000000000 5000000000 false 0xc000094a50 loki false { false}} 0xc000094a50 loki [{ringDesc 0xe43cc0} {}]} 0xc00012b720 0xc000094a50 0xc0007688c0 {{} [0 1 0]} {0 0} map map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map map {0 0} 0 0 0 {0 0} map 0xc00023a540 0xc0006e13e0 0xc0006e14a0 0xc0006e1560 0xc0006e1620 0xc0006e1860 0xc0006e16e0 0xc0006e1920 0xc0006e17a0 0xc00060e8c0 0xc00060e900 0xc0006e1aa0 0xc0006e1b60 0xc0006e1ce0 0xc0006e1c20 0xc00000e5b0 0xc000769500 0xc00000e5a0 0xc00000e5a8 0xc00060e940 0xc00060e980 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided”

level=info ts=2023-08-28T08:02:09.972266973Z caller=modules.go:1090 msg=“server stopped”

level=info ts=2023-08-28T08:02:09.972284681Z caller=module_service.go:114 msg=“module stopped” module=server

level=info ts=2023-08-28T08:02:09.972299161Z caller=loki.go:375 msg=“Loki stopped”

failed services

github.com/grafana/loki/pkg/loki.(*Loki).Run

/src/loki/pkg/loki/loki.go:421

main.main

/src/loki/cmd/loki/main.go:105

runtime.main

/usr/local/go/src/runtime/proc.go:255

runtime.goexit

/usr/local/go/src/runtime/asm_amd64.s:1581

level=error ts=2023-08-28T08:02:09.972391639Z caller=log.go:103 msg=“error running loki” err=“failed services\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:421\nmain.main\n\t/src/loki/cmd/loki/main.go:105\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581”

please help me to resolve this issue

Hi @ashishsharma1699 ,
It would help to know how you installed Loki.

E.g. if you used a Helm chart

  • which Helm chart (there are several)
  • please share your values.yaml

If you did not use Helm, a copy of your Loki config.yaml would be helpful.

@b0b I’m sharing the helm chart and values with loki. You can refer to blog which I used to setup loki.

Please check. it seems issue with image and actual codebase.

I am using command

helm upgrade --install loki grafana/loki-stack --namespace=monitoring --set grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=true,prometheus.server.persistentVolume.enabled=false,loki.persistence.enabled=true,loki.persistence.storageClassName=gp2,loki.persistence.size=5Gi,grafana.persistence.enabled=true,grafana.persistence.storageClassName=gp2,grafana.persistence.size=5Gi

only loki pod is failing and other pods runs successfully

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.