Trying to setup Simple Scalable loki using instructions provided at Install the simple scalable Helm chart | Grafana Loki documentation but few containers are in CrashLoopBackOff. Using minikube to setup 4 nodes k8s cluster using docker driver. Post deploy, loki-backend continues to fail, error below:
{"time": "2025-03-14T17:14:55.166773+00:00", "level": "INFO", "msg": "Starting collector"}
{"time": "2025-03-14T17:14:55.167211+00:00", "level": "WARNING", "msg": "No folder annotation was provided, defaulting to k8s-sidecar-target-directory"}
{"time": "2025-03-14T17:14:55.167536+00:00", "level": "INFO", "msg": "Loading incluster config ..."}
{"time": "2025-03-14T17:14:55.168626+00:00", "level": "INFO", "msg": "Config for cluster api at 'https://10.96.0.1:443' loaded..."}
{"time": "2025-03-14T17:14:55.168857+00:00", "level": "INFO", "msg": "Unique filenames will not be enforced."}
{"time": "2025-03-14T17:14:55.169045+00:00", "level": "INFO", "msg": "5xx response content will not be enabled."}
loki-write components continue to fail due to:
mkdir /var/loki/tsdb-shipper-active: permission denied
error initialising module: store
github.com/grafana/dskit/modules.(*Manager).initModule
/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108
github.com/grafana/loki/v3/pkg/loki.(*Loki).Run
/src/loki/pkg/loki/loki.go:495
main.main
/src/loki/cmd/loki/main.go:129
runtime.main
/usr/local/go/src/runtime/proc.go:272
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1700
level=info ts=2025-03-14T19:02:07.424455349Z caller=main.go:126 msg="Starting Loki" version="(version=3.4.2, branch=release-3.4.x, revision=4fa045d3)"
level=info ts=2025-03-14T19:02:07.424497684Z caller=main.go:127 msg="Loading configuration file" filename=/etc/loki/config/config.yaml
level=info ts=2025-03-14T19:02:07.429396054Z caller=server.go:351 msg="server listening on addresses" http=[::]:3100 grpc=[::]:9095
level=info ts=2025-03-14T19:02:07.445468405Z caller=memberlist_client.go:439 msg="Using memberlist cluster label and node name" cluster_label= node=loki-write-2-80419bb3
level=info ts=2025-03-14T19:02:07.446476443Z caller=memberlist_client.go:549 msg="memberlist fast-join starting" nodes_found=1 to_join=4
level=error ts=2025-03-14T19:02:07.451518056Z caller=resolver.go:100 msg="failed to lookup SRV records" host=_memcached-client._tcp.loki-chunks-cache.default.svc err="lookup _memcached-client._tcp.loki-chunks-cache.default.svc on 10.96.0.10:53: no such host"
level=info ts=2025-03-14T19:02:07.45987262Z caller=shipper.go:165 index-store=tsdb-2024-04-01 msg="starting index shipper in WO mode"
level=error ts=2025-03-14T19:02:07.460783873Z caller=log.go:216 msg="error running loki" err="mkdir /var/loki/tsdb-shipper-active: permission denied\nerror initialising module: store\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:495\nmain.main\n\t/src/loki/cmd/loki/main.go:129\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"
Overall status:
NAME READY STATUS RESTARTS AGE
loki-backend-0 1/2 CrashLoopBackOff 26 (66s ago) 110m
loki-backend-1 1/2 CrashLoopBackOff 26 (19s ago) 110m
loki-backend-2 1/2 CrashLoopBackOff 26 (101s ago) 110m
loki-canary-5n5q7 1/1 Running 0 110m
loki-canary-8mrzw 1/1 Running 0 110m
loki-canary-f5cgr 1/1 Running 0 110m
loki-canary-lrzpg 1/1 Running 0 110m
loki-chunks-cache-0 0/2 Pending 0 110m
loki-gateway-5858bd5949-vz47x 1/1 Running 0 110m
loki-minio-0 0/1 CrashLoopBackOff 26 (69s ago) 110m
loki-read-5df6cf8c68-9h27r 0/1 Running 0 110m
loki-read-5df6cf8c68-mktw6 0/1 Running 0 110m
loki-read-5df6cf8c68-szsmq 0/1 Running 0 110m
loki-results-cache-0 2/2 Running 0 110m
loki-write-0 0/1 CrashLoopBackOff 26 (102s ago) 110m
loki-write-1 0/1 CrashLoopBackOff 26 (12s ago) 110m
loki-write-2 0/1 CrashLoopBackOff 26 (79s ago) 110m
Any recommendations how to get the deployment to work? Thanks.