Loki 2.6.1 not starting in Helm deployment

Trying to run Loki through the current version of Helm deployment. Startup of the loki pod is failing with the error message:

msg=“module failed” module=memberlist-kv error=“invalid service state: Stopping, expected: Running”

Helm deployment was done through:

helm upgrade --install loki --namespace=loki --create-namespace grafana/loki-stack

Here is the full log I am seeing:

level=info ts=2022-10-26T13:35:25.850991579Z caller=main.go:103 msg=“Starting Loki” version=“(version=2.6.1, branch=HEAD, revision=6bd05c9a4)”
level=info ts=2022-10-26T13:35:25.851328801Z caller=server.go:288 http=[::]:3100 grpc=[::]:9095 msg=“server listening on addresses”
level=info ts=2022-10-26T13:35:25.851583817Z caller=modules.go:736 msg=“RulerStorage is not configured in single binary mode and will not be started.”
level=warn ts=2022-10-26T13:35:25.85238377Z caller=experimental.go:20 msg=“experimental feature in use” feature=“In-memory (FIFO) cache - chunksfifocache”
level=info ts=2022-10-26T13:35:25.85269369Z caller=table_manager.go:252 msg=“query readiness setup completed” duration=2.1µs distinct_users_len=0
level=info ts=2022-10-26T13:35:25.852726292Z caller=shipper.go:124 msg=“starting index shipper in RW mode”
level=info ts=2022-10-26T13:35:25.852811798Z caller=shipper_index_client.go:79 msg=“starting boltdb shipper in RW mode”
level=info ts=2022-10-26T13:35:25.854365199Z caller=table_manager.go:134 msg=“uploading tables”
level=info ts=2022-10-26T13:35:25.854403001Z caller=table_manager.go:167 msg=“handing over indexes to shipper”
level=info ts=2022-10-26T13:35:25.856349928Z caller=modules.go:761 msg=“RulerStorage is nil. Not starting the ruler.”
level=info ts=2022-10-26T13:35:25.856933966Z caller=worker.go:112 msg=“Starting querier worker using query-scheduler and scheduler ring for addresses”
level=info ts=2022-10-26T13:35:25.859298021Z caller=module_service.go:82 msg=initialising module=server
level=info ts=2022-10-26T13:35:25.859395127Z caller=module_service.go:82 msg=initialising module=memberlist-kv
level=info ts=2022-10-26T13:35:25.859398327Z caller=module_service.go:82 msg=initialising module=query-frontend-tripperware
level=error ts=2022-10-26T13:35:25.85943933Z caller=loki.go:386 msg=“module failed” module=memberlist-kv error=“invalid service state: Stopping, expected: Running”
level=error ts=2022-10-26T13:35:25.859471132Z caller=loki.go:386 msg=“module failed” module=query-scheduler error=“failed to start query-scheduler, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running”
level=error ts=2022-10-26T13:35:25.859485233Z caller=loki.go:386 msg=“module failed” module=store error=“failed to start store, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running”
level=error ts=2022-10-26T13:35:25.859494133Z caller=loki.go:386 msg=“module failed” module=distributor error=“failed to start distributor, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running”
level=error ts=2022-10-26T13:35:25.859502434Z caller=loki.go:386 msg=“module failed” module=ingester error=“failed to start ingester, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running”
level=info ts=2022-10-26T13:35:25.859501834Z caller=module_service.go:114 msg=“module stopped” module=query-frontend-tripperware
level=error ts=2022-10-26T13:35:25.859510535Z caller=loki.go:386 msg=“module failed” module=ring error=“failed to start ring, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running”
level=error ts=2022-10-26T13:35:25.859518235Z caller=loki.go:386 msg=“module failed” module=compactor error=“failed to start compactor, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running”
level=error ts=2022-10-26T13:35:25.859528236Z caller=loki.go:386 msg=“module failed” module=ingester-querier error=“failed to start ingester-querier, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running”
level=error ts=2022-10-26T13:35:25.859537236Z caller=loki.go:386 msg=“module failed” module=usage-report error=“failed to start usage-report, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running”
level=error ts=2022-10-26T13:35:25.859545137Z caller=loki.go:386 msg=“module failed” module=query-frontend error=“failed to start query-frontend, because it depends on module usage-report, which has failed: context canceled”
level=error ts=2022-10-26T13:35:25.859559138Z caller=loki.go:386 msg=“module failed” module=querier error=“failed to start querier, because it depends on module query-scheduler, which has failed: context canceled”
level=info ts=2022-10-26T13:35:25.860760616Z caller=modules.go:1090 msg=“server stopped”
level=info ts=2022-10-26T13:35:25.860780417Z caller=module_service.go:114 msg=“module stopped” module=server
level=info ts=2022-10-26T13:35:25.860792318Z caller=loki.go:375 msg=“Loki stopped”
failed services
github.com/grafana/loki/pkg/loki.(*Loki).Run
/src/loki/pkg/loki/loki.go:421
main.main
/src/loki/cmd/loki/main.go:105
runtime.main
/usr/local/go/src/runtime/proc.go:255
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1581
level=error ts=2022-10-26T13:35:25.860839221Z caller=log.go:103 msg=“error running loki” err=“failed services\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:421\nmain.main\n\t/src/loki/cmd/loki/main.go:105\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581”

Any ideas anyone?

Rudy

Same issue with my setup. Any fix for it?

1 Like

Same issue with k8s : 1.27 , Any solution for this ?