I’m trying to configure Loki in simple-scalable fashion. I have Istio enabled too. I also have auth (multi-tenant enabled)
When calling Loki via Grafana, getting this:
With this Loki config:
# https://grafana.com/docs/loki/latest/configuration/
auth_enabled: true # X-Scope-OrgID
common:
path_prefix: /var/loki
replication_factor: 1
ring:
kvstore:
store: memberlist
storage:
filesystem:
chunks_directory: /var/loki/chunks
rules_directory: /var/loki/rules
limits_config:
enforce_metric_name: false
max_cache_freshness_per_query: 10m
reject_old_samples: true
reject_old_samples_max_age: 168h
memberlist:
join_members:
- dnssrv+loki-memberlist.observability.svc
compactor:
working_directory: /var/loki/compactor
shared_store: gcs
# https://grafana.com/docs/loki/latest/configuration/#storage_config
storage_config:
boltdb_shipper:
active_index_directory: /var/loki/boltdb-shipper-active
cache_location: /var/loki/boltdb-shipper-cache
cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space
shared_store: gcs
# index_gateway_client:
# server_address: loki-gateway.observability.svc
gcs:
bucket_name: loki-boltdb-shipper
schema_config:
configs:
- from: "2020-07-01"
store: boltdb-shipper
object_store: gcs
schema: v11
index:
prefix: index_
period: 24h
# tracing
# compactor
# ingester
# ruler
server:
http_listen_port: 3100
With this Grafana config:
With logs on the reader:
loki-reader-0 loki level=debug ts=2022-01-21T15:33:29.295122172Z caller=grpc_logging.go:41 duration=77.349µs method=/frontendv2pb.FrontendForQuerier/QueryResult msg="gRPC (success)"
loki-reader-0 loki level=error ts=2022-01-21T15:33:29.295365949Z caller=retry.go:73 org_id=CQ-15396174 msg="error processing request" try=0 err="rpc error: code = Code(500) desc = empty ring\n"
loki-reader-0 loki level=debug ts=2022-01-21T15:33:29.296433541Z caller=grpc_logging.go:41 method=/frontendv2pb.FrontendForQuerier/QueryResult duration=54.031µs msg="gRPC (success)"
loki-reader-0 loki level=error ts=2022-01-21T15:33:29.296546223Z caller=retry.go:73 org_id=CQ-15396174 msg="error processing request" try=1 err="rpc error: code = Code(500) desc = empty ring\n"
loki-reader-0 loki level=debug ts=2022-01-21T15:33:29.29741856Z caller=grpc_logging.go:41 method=/frontendv2pb.FrontendForQuerier/QueryResult duration=45.776µs msg="gRPC (success)"
loki-reader-0 loki level=error ts=2022-01-21T15:33:29.297558665Z caller=retry.go:73 org_id=CQ-15396174 msg="error processing request" try=2 err="rpc error: code = Code(500) desc = empty ring\n"
loki-reader-0 loki level=debug ts=2022-01-21T15:33:29.29868191Z caller=grpc_logging.go:41 method=/frontendv2pb.FrontendForQuerier/QueryResult duration=120.278µs msg="gRPC (success)"
loki-reader-0 loki level=error ts=2022-01-21T15:33:29.298802582Z caller=retry.go:73 org_id=CQ-15396174 msg="error processing request" try=3 err="rpc error: code = Code(500) desc = empty ring\n"
loki-reader-0 loki level=debug ts=2022-01-21T15:33:29.299697Z caller=grpc_logging.go:41 method=/frontendv2pb.FrontendForQuerier/QueryResult duration=49.121µs msg="gRPC (success)"
loki-reader-0 loki level=error ts=2022-01-21T15:33:29.299784206Z caller=retry.go:73 org_id=CQ-15396174 msg="error processing request" try=4 err="rpc error: code = Code(500) desc = empty ring\n"
loki-reader-0 loki level=warn ts=2022-01-21T15:33:29.299956808Z caller=logging.go:72 traceID=013d77074bc6e39f orgID=CQ-15396174 msg="GET /loki/api/v1/label?start=1642778609238000000 (500) 7.823636ms Response: \"empty ring\\n\" ws: false; Accept: application/json, text/plain, */*; Accept-Encoding: gzip; Accept-Language: en-GB,en;q=0.7,sv;q=0.3; Cdn-Loop: cloudflare; Cf-Connecting-Ip: 83.241.140.206; Cf-Ipcountry: SE; Cf-Ray: 6d1199e9c89c7264-HAM; Cf-Visitor: {\"scheme\":\"https\"}; Dnt: 1; Sec-Fetch-Dest: empty; Sec-Fetch-Mode: cors; Sec-Fetch-Site: same-origin; User-Agent: Grafana/8.3.3; X-B3-Parentspanid: c5d0f9ffbd508ffa; X-B3-Sampled: 1; X-B3-Spanid: f3646f68f02c4050; X-B3-Traceid: f8b77e85143cc315293f82425e365143; X-Envoy-Attempt-Count: 1; X-Envoy-External-Address: 162.158.202.235; X-Forwarded-Client-Cert: By=spiffe://cluster.local/ns/observability/sa/loki;Hash=92cccf37a8ef35811f6ff83afc1b3761edb7fc0e22f403a084462ed0c3a1eae0;Subject=\"\";URI=spiffe://cluster.local/ns/observability/sa/grafana; X-Forwarded-For: 83.241.140.206,162.158.202.235, 127.0.0.6, 127.0.0.6; X-Forwarded-Proto: http; X-Grafana-Nocache: true; X-Grafana-Org-Id: 1; X-Request-Id: 7bd06f15-acb9-99b8-997e-fa3508815185; X-Scope-Orgid: CQ-15396174; "
loki-reader-0 loki level=debug ts=2022-01-21T15:33:31.504260244Z caller=grpc_logging.go:41 method=/grpc.health.v1.Health/Check duration=60.188µs msg="gRPC (success)"
$ stern loki-writer
+ loki-writer-0 › loki
+ loki-writer-0 › istio-proxy
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.767749Z","msg":"FLAG: --concurrency=\"2\""}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.767778Z","msg":"FLAG: --domain=\"observability.svc.cluster.local\""}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.767785Z","msg":"FLAG: --help=\"false\""}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.767789Z","msg":"FLAG: --log_as_json=\"true\""}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.767792Z","msg":"FLAG: --log_caller=\"\""}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.767796Z","msg":"FLAG: --log_output_level=\"default:info\""}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.767800Z","msg":"FLAG: --log_rotate=\"\""}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.767803Z","msg":"FLAG: --log_rotate_max_age=\"30\""}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.767807Z","msg":"FLAG: --log_rotate_max_backups=\"1000\""}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.767811Z","msg":"FLAG: --log_rotate_max_size=\"104857600\""}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.767815Z","msg":"FLAG: --log_stacktrace_level=\"default:none\""}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.767824Z","msg":"FLAG: --log_target=\"[stdout]\""}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.767838Z","msg":"FLAG: --meshConfig=\"./etc/istio/config/mesh\""}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.767841Z","msg":"FLAG: --outlierLogPath=\"\""}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.767845Z","msg":"FLAG: --proxyComponentLogLevel=\"misc:error\""}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.767848Z","msg":"FLAG: --proxyLogLevel=\"warning\""}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.767852Z","msg":"FLAG: --serviceCluster=\"loki.observability\""}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.767856Z","msg":"FLAG: --stsPort=\"0\""}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.767859Z","msg":"FLAG: --templateFile=\"\""}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.767862Z","msg":"FLAG: --tokenManagerPlugin=\"GoogleTokenExchange\""}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.767866Z","msg":"Version 1.10.0-d26cba7e341587453ffeb978f5cf6fbc32f346f8-Clean"}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.768090Z","msg":"Proxy role","ips":["10.4.14.146"],"type":"sidecar","id":"loki-writer-0.observability","domain":"observability.svc.cluster.local"}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.768205Z","msg":"Apply proxy config from env {\"tracing\":{\"zipkin\":{\"address\":\"jaeger-collector.observability.svc.cluster.local:9411\"},\"sampling\":100}}\n"}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.769281Z","msg":"Effective config: binaryPath: /usr/local/bin/envoy\nconcurrency: 2\nconfigPath: ./etc/istio/proxy\ncontrolPlaneAuthPolicy: MUTUAL_TLS\ndiscoveryAddress: istiod.istio-system.svc:15012\ndrainDuration: 45s\nparentShutdownDuration: 60s\nproxyAdminPort: 15000\nserviceCluster: loki.observability\nstatNameLength: 189\nstatusPort: 15020\nterminationDrainDuration: 5s\ntracing:\n sampling: 100\n zipkin:\n address: jaeger-collector.observability.svc.cluster.local:9411\n"}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.769318Z","msg":"JWT policy is third-party-jwt"}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.769339Z","msg":"Pilot SAN: [istiod.istio-system.svc]"}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.769344Z","msg":"CA Endpoint istiod.istio-system.svc:15012, provider Citadel"}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.769393Z","msg":"Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem"}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.769511Z","scope":"citadelclient","msg":"Citadel client using custom root cert: istiod.istio-system.svc:15012"}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.807751Z","scope":"ads","msg":"All caches have been synced up in 43.551911ms, marking server ready"}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.808137Z","scope":"sds","msg":"SDS server for workload certificates started, listening on \"./etc/istio/proxy/SDS\""}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.808173Z","scope":"xdsproxy","msg":"Initializing with upstream address \"istiod.istio-system.svc:15012\" and cluster \"Kubernetes\""}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.808177Z","scope":"sds","msg":"Start SDS grpc server"}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.808589Z","msg":"Opening status port 15020"}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.995904Z","scope":"cache","msg":"generated new workload certificate","latency":187746704,"ttl":86399004108654}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.995968Z","scope":"cache","msg":"Root cert has changed, start rotating root cert"}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.995994Z","scope":"ads","msg":"XDS: Incremental Pushing:0 ConnectedEndpoints:0 Version:"}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:42.996039Z","scope":"cache","msg":"returned workload trust anchor from cache","ttl":86399003963110}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:43.819747Z","msg":"Starting proxy agent"}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:43.819783Z","msg":"Epoch 0 starting"}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:43.822178Z","msg":"Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --drain-strategy immediate --parent-shutdown-time-s 60 --service-cluster loki.observability --service-node sidecar~10.4.14.146~loki-writer-0.observability~observability.svc.cluster.local --local-address-ip-version v4 --bootstrap-version 3 --disable-hot-restart --log-format {\"level\":\"%l\",\"time\":\"%Y-%m-%dT%T.%fZ\",\"scope\":\"envoy %n\",\"msg\":\"%j\"} -l warning --component-log-level misc:error --concurrency 2]"}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:43.916157Z","scope":"xdsproxy","msg":"connected to upstream XDS server: istiod.istio-system.svc:15012"}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:43.989461Z","scope":"ads","msg":"ADS: new connection for node:sidecar~10.4.14.146~loki-writer-0.observability~observability.svc.cluster.local-1"}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:43.989560Z","scope":"cache","msg":"returned workload trust anchor from cache","ttl":86398010443209}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:43.989631Z","scope":"ads","msg":"ADS: new connection for node:sidecar~10.4.14.146~loki-writer-0.observability~observability.svc.cluster.local-2"}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:43.989933Z","scope":"sds","msg":"SDS: PUSH","resource":"ROOTCA"}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:43.989995Z","scope":"cache","msg":"returned workload certificate from cache","ttl":86398010010174}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:43.990241Z","scope":"sds","msg":"SDS: PUSH","resource":"default"}
loki-writer-0 istio-proxy {"response_flags":"-","protocol":"HTTP/1.1","path":"/computeMetadata/v1/project/project-id","route_name":"default","upstream_host":"169.254.169.254:80","response_code_details":"via_upstream","upstream_service_time":"0","downstream_remote_address":"10.4.14.146:33478","duration":0,"user_agent":"gcloud-golang/0.1","requested_server_name":null,"downstream_local_address":"169.254.169.254:80","method":"GET","upstream_local_address":"10.4.14.146:33480","upstream_transport_failure_reason":null,"start_time":"2022-01-21T15:31:45.103Z","response_code":200,"request_id":"c53c7f35-edb9-9797-b138-9a19fc8c1446","authority":"169.254.169.254","x_forwarded_for":null,"bytes_received":0,"connection_termination_details":null,"upstream_cluster":"outbound|80||metadata.google.internal","bytes_sent":11}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:45.417722Z","msg":"Initialization took 2.652426033s"}
loki-writer-0 istio-proxy {"level":"info","time":"2022-01-21T15:31:45.417757Z","msg":"Envoy proxy is ready"}
loki-writer-0 loki 2022-01-21 15:31:42.614107 I | proto: duplicate proto type registered: purgeplan.DeletePlan
loki-writer-0 loki 2022-01-21 15:31:42.614161 I | proto: duplicate proto type registered: purgeplan.ChunksGroup
loki-writer-0 loki 2022-01-21 15:31:42.614167 I | proto: duplicate proto type registered: purgeplan.ChunkDetails
loki-writer-0 loki 2022-01-21 15:31:42.614171 I | proto: duplicate proto type registered: purgeplan.Interval
loki-writer-0 loki 2022-01-21 15:31:42.630289 I | proto: duplicate proto type registered: grpc.PutChunksRequest
loki-writer-0 loki 2022-01-21 15:31:42.630308 I | proto: duplicate proto type registered: grpc.GetChunksRequest
loki-writer-0 loki 2022-01-21 15:31:42.630313 I | proto: duplicate proto type registered: grpc.GetChunksResponse
loki-writer-0 loki 2022-01-21 15:31:42.630317 I | proto: duplicate proto type registered: grpc.Chunk
loki-writer-0 loki 2022-01-21 15:31:42.630321 I | proto: duplicate proto type registered: grpc.ChunkID
loki-writer-0 loki 2022-01-21 15:31:42.630325 I | proto: duplicate proto type registered: grpc.DeleteTableRequest
loki-writer-0 loki 2022-01-21 15:31:42.630330 I | proto: duplicate proto type registered: grpc.DescribeTableRequest
loki-writer-0 loki 2022-01-21 15:31:42.630334 I | proto: duplicate proto type registered: grpc.WriteBatch
loki-writer-0 loki 2022-01-21 15:31:42.630338 I | proto: duplicate proto type registered: grpc.WriteIndexRequest
loki-writer-0 loki 2022-01-21 15:31:42.630341 I | proto: duplicate proto type registered: grpc.DeleteIndexRequest
loki-writer-0 loki 2022-01-21 15:31:42.630345 I | proto: duplicate proto type registered: grpc.QueryIndexResponse
loki-writer-0 loki 2022-01-21 15:31:42.630349 I | proto: duplicate proto type registered: grpc.Row
loki-writer-0 loki 2022-01-21 15:31:42.630364 I | proto: duplicate proto type registered: grpc.IndexEntry
loki-writer-0 loki 2022-01-21 15:31:42.630374 I | proto: duplicate proto type registered: grpc.QueryIndexRequest
loki-writer-0 loki 2022-01-21 15:31:42.630379 I | proto: duplicate proto type registered: grpc.UpdateTableRequest
loki-writer-0 loki 2022-01-21 15:31:42.630384 I | proto: duplicate proto type registered: grpc.DescribeTableResponse
loki-writer-0 loki 2022-01-21 15:31:42.630389 I | proto: duplicate proto type registered: grpc.CreateTableRequest
loki-writer-0 loki 2022-01-21 15:31:42.630394 I | proto: duplicate proto type registered: grpc.TableDesc
loki-writer-0 loki 2022-01-21 15:31:42.630399 I | proto: duplicate proto type registered: grpc.TableDesc.TagsEntry
loki-writer-0 loki 2022-01-21 15:31:42.630403 I | proto: duplicate proto type registered: grpc.ListTablesResponse
loki-writer-0 loki 2022-01-21 15:31:42.630408 I | proto: duplicate proto type registered: grpc.Labels
loki-writer-0 loki 2022-01-21 15:31:42.630542 I | proto: duplicate proto type registered: storage.Entry
loki-writer-0 loki 2022-01-21 15:31:42.630551 I | proto: duplicate proto type registered: storage.ReadBatch
loki-writer-0 loki level=info ts=2022-01-21T15:31:42.637105114Z caller=main.go:94 msg="Starting Loki" version="(version=2.4.2, branch=HEAD, revision=525040a32)"
loki-writer-0 loki level=info ts=2022-01-21T15:31:42.6374589Z caller=server.go:260 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
loki-writer-0 loki level=warn ts=2022-01-21T15:31:42.638944112Z caller=experimental.go:19 msg="experimental feature in use" feature="In-memory (FIFO) cache"
loki-writer-0 loki level=warn ts=2022-01-21T15:31:42.63914851Z caller=experimental.go:19 msg="experimental feature in use" feature="In-memory (FIFO) cache"
loki-writer-0 loki level=info ts=2022-01-21T15:31:42.639536871Z caller=memberlist_client.go:394 msg="Using memberlist cluster node name" name=loki-writer-0-784efbd1
loki-writer-0 loki ts=2022-01-21T15:31:42.63965871Z caller=memberlist_logger.go:74 level=debug msg="configured Transport is not a NodeAwareTransport and some features may not work as desired"
loki-writer-0 loki level=debug ts=2022-01-21T15:31:42.640454764Z caller=tcp_transport.go:389 msg=FinalAdvertiseAddr advertiseAddr=10.4.14.146 advertisePort=7946
loki-writer-0 loki level=debug ts=2022-01-21T15:31:42.641030907Z caller=tcp_transport.go:389 msg=FinalAdvertiseAddr advertiseAddr=10.4.14.146 advertisePort=7946
loki-writer-0 loki level=error ts=2022-01-21T15:31:42.6755985Z caller=resolver.go:99 msg="failed to lookup SRV records" host=loki-memberlist.observability.svc err="lookup loki-memberlist.observability.svc on 172.16.16.10:53: no such host"
loki-writer-0 loki level=info ts=2022-01-21T15:31:42.675682193Z caller=memberlist_client.go:506 msg="joined memberlist cluster" reached_nodes=0
loki-writer-0 loki level=info ts=2022-01-21T15:31:43.189577537Z caller=shipper_index_client.go:108 msg="starting boltdb shipper in 2 mode"
loki-writer-0 loki level=info ts=2022-01-21T15:31:43.189694187Z caller=table_manager.go:171 msg="uploading tables"
loki-writer-0 loki level=debug ts=2022-01-21T15:31:45.124518369Z caller=module_service.go:54 msg="module waiting for initialization" module=ingester waiting_for=memberlist-kv
loki-writer-0 loki level=info ts=2022-01-21T15:31:45.124584901Z caller=module_service.go:64 msg=initialising module=store
loki-writer-0 loki level=info ts=2022-01-21T15:31:45.124610713Z caller=module_service.go:64 msg=initialising module=server
loki-writer-0 loki level=debug ts=2022-01-21T15:31:45.124598267Z caller=module_service.go:54 msg="module waiting for initialization" module=ring waiting_for=memberlist-kv
loki-writer-0 loki level=debug ts=2022-01-21T15:31:45.124646783Z caller=module_service.go:54 msg="module waiting for initialization" module=distributor waiting_for=memberlist-kv
loki-writer-0 loki level=info ts=2022-01-21T15:31:45.124649459Z caller=module_service.go:64 msg=initialising module=memberlist-kv
loki-writer-0 loki level=debug ts=2022-01-21T15:31:45.124697814Z caller=module_service.go:54 msg="module waiting for initialization" module=distributor waiting_for=ring
loki-writer-0 loki level=debug ts=2022-01-21T15:31:45.124699145Z caller=module_service.go:54 msg="module waiting for initialization" module=ingester waiting_for=server
loki-writer-0 loki level=debug ts=2022-01-21T15:31:45.124714614Z caller=module_service.go:54 msg="module waiting for initialization" module=ingester waiting_for=store
loki-writer-0 loki level=info ts=2022-01-21T15:31:45.124723521Z caller=module_service.go:64 msg=initialising module=ingester
loki-writer-0 loki level=debug ts=2022-01-21T15:31:45.124722191Z caller=module_service.go:54 msg="module waiting for initialization" module=ring waiting_for=server
loki-writer-0 loki level=info ts=2022-01-21T15:31:45.124735194Z caller=module_service.go:64 msg=initialising module=ring
loki-writer-0 loki level=info ts=2022-01-21T15:31:45.12474436Z caller=ingester.go:353 msg="recovering from checkpoint"
loki-writer-0 loki level=info ts=2022-01-21T15:31:45.12491278Z caller=ring.go:281 msg="ring doesn't exist in KV store yet"
loki-writer-0 loki level=debug ts=2022-01-21T15:31:45.124994037Z caller=module_service.go:54 msg="module waiting for initialization" module=distributor waiting_for=server
loki-writer-0 loki level=info ts=2022-01-21T15:31:45.125006771Z caller=module_service.go:64 msg=initialising module=distributor
loki-writer-0 loki level=info ts=2022-01-21T15:31:45.12506792Z caller=lifecycler.go:540 msg="not loading tokens from file, tokens file path is empty"
loki-writer-0 loki level=info ts=2022-01-21T15:31:45.125109213Z caller=lifecycler.go:569 msg="instance not found in ring, adding with no tokens" ring=distributor
loki-writer-0 loki level=debug ts=2022-01-21T15:31:45.125930508Z caller=lifecycler.go:412 msg="JoinAfter expired" ring=distributor
loki-writer-0 loki level=info ts=2022-01-21T15:31:45.125971562Z caller=lifecycler.go:416 msg="auto-joining cluster after timeout" ring=distributor
loki-writer-0 loki level=debug ts=2022-01-21T15:31:45.126533784Z caller=memberlist_client.go:814 msg="CAS attempt failed" err="no change detected" retry=true
loki-writer-0 loki level=info ts=2022-01-21T15:31:45.167513334Z caller=ingester.go:369 msg="recovered WAL checkpoint recovery finished" elapsed=42.775766ms errors=false
loki-writer-0 loki level=info ts=2022-01-21T15:31:45.167566047Z caller=ingester.go:375 msg="recovering from WAL"
loki-writer-0 loki level=info ts=2022-01-21T15:31:45.167869252Z caller=ingester.go:391 msg="WAL segment recovery finished" elapsed=43.13381ms errors=false
loki-writer-0 loki level=info ts=2022-01-21T15:31:45.167915624Z caller=ingester.go:339 msg="closing recoverer"
loki-writer-0 loki level=info ts=2022-01-21T15:31:45.167931605Z caller=ingester.go:347 msg="WAL recovery finished" time=43.196732ms
loki-writer-0 loki level=info ts=2022-01-21T15:31:45.168009014Z caller=lifecycler.go:540 msg="not loading tokens from file, tokens file path is empty"
loki-writer-0 loki level=info ts=2022-01-21T15:31:45.167983654Z caller=wal.go:156 msg=started component=wal
loki-writer-0 loki level=info ts=2022-01-21T15:31:45.168070203Z caller=loki.go:355 msg="Loki started"
loki-writer-0 loki level=info ts=2022-01-21T15:31:45.168087615Z caller=lifecycler.go:569 msg="instance not found in ring, adding with no tokens" ring=ingester
loki-writer-0 loki level=debug ts=2022-01-21T15:31:45.168278527Z caller=lifecycler.go:412 msg="JoinAfter expired" ring=ingester
loki-writer-0 loki level=info ts=2022-01-21T15:31:45.168323352Z caller=lifecycler.go:416 msg="auto-joining cluster after timeout" ring=ingester
loki-writer-0 loki level=debug ts=2022-01-21T15:31:45.168669105Z caller=memberlist_client.go:814 msg="CAS attempt failed" err="no change detected" retry=true
loki-writer-0 loki level=debug ts=2022-01-21T15:31:46.127747637Z caller=broadcast.go:48 msg="Invalidating forwarded broadcast" key=collectors/distributor version=2 oldVersion=1 content=[loki-writer-0] oldContent=[loki-writer-0]
How would I solve this?