"panic: runtime error: invalid memory address or nil pointer dereference" with main-93507a0 docker images

Hi,
I’ve used grafana-loki in a docker container since March 2021 and in the last months I did not change the configuration while I’ve kept the docker image always up to date with an automatic update (with watchtower container) once a week. Yesterday it got updated and started failing as soon as it is started and it still fails with today latest image (which is main-93507a0).

Container logs shows the following:

2021-10-23 18:20:43.637722 I | proto: duplicate proto type registered: purgeplan.DeletePlan
2021-10-23 18:20:43.637799 I | proto: duplicate proto type registered: purgeplan.ChunksGroup
2021-10-23 18:20:43.637817 I | proto: duplicate proto type registered: purgeplan.ChunkDetails
2021-10-23 18:20:43.637836 I | proto: duplicate proto type registered: purgeplan.Interval
2021-10-23 18:20:43.657803 I | proto: duplicate proto type registered: grpc.PutChunksRequest
2021-10-23 18:20:43.657850 I | proto: duplicate proto type registered: grpc.GetChunksRequest
2021-10-23 18:20:43.657865 I | proto: duplicate proto type registered: grpc.GetChunksResponse
2021-10-23 18:20:43.657880 I | proto: duplicate proto type registered: grpc.Chunk
2021-10-23 18:20:43.657894 I | proto: duplicate proto type registered: grpc.ChunkID
2021-10-23 18:20:43.657907 I | proto: duplicate proto type registered: grpc.DeleteTableRequest
2021-10-23 18:20:43.657922 I | proto: duplicate proto type registered: grpc.DescribeTableRequest
2021-10-23 18:20:43.657937 I | proto: duplicate proto type registered: grpc.WriteBatch
2021-10-23 18:20:43.657950 I | proto: duplicate proto type registered: grpc.WriteIndexRequest
2021-10-23 18:20:43.658005 I | proto: duplicate proto type registered: grpc.DeleteIndexRequest
2021-10-23 18:20:43.658019 I | proto: duplicate proto type registered: grpc.QueryIndexResponse
2021-10-23 18:20:43.658032 I | proto: duplicate proto type registered: grpc.Row
2021-10-23 18:20:43.658045 I | proto: duplicate proto type registered: grpc.IndexEntry
2021-10-23 18:20:43.658058 I | proto: duplicate proto type registered: grpc.QueryIndexRequest
2021-10-23 18:20:43.658080 I | proto: duplicate proto type registered: grpc.UpdateTableRequest
2021-10-23 18:20:43.658094 I | proto: duplicate proto type registered: grpc.DescribeTableResponse
2021-10-23 18:20:43.658161 I | proto: duplicate proto type registered: grpc.CreateTableRequest
2021-10-23 18:20:43.658177 I | proto: duplicate proto type registered: grpc.TableDesc
2021-10-23 18:20:43.658191 I | proto: duplicate proto type registered: grpc.TableDesc.TagsEntry
2021-10-23 18:20:43.658204 I | proto: duplicate proto type registered: grpc.ListTablesResponse
2021-10-23 18:20:43.658217 I | proto: duplicate proto type registered: grpc.Labels
2021-10-23 18:20:43.659616 I | proto: duplicate proto type registered: storage.Entry
2021-10-23 18:20:43.659637 I | proto: duplicate proto type registered: storage.ReadBatch
level=info ts=2021-10-23T18:20:43.669478726Z caller=main.go:94 msg="Starting Loki" version="(version=main-e3726a1, branch=main, revision=e3726a19c)"
level=info ts=2021-10-23T18:20:43.670060286Z caller=server.go:260 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2021-10-23T18:20:43.671489646Z caller=table_manager.go:241 msg="loading table index_18923"
level=info ts=2021-10-23T18:20:44.271364121Z caller=table.go:335 msg="uploading table index_18923"
level=info ts=2021-10-23T18:20:44.271523669Z caller=shipper_index_client.go:108 msg="starting boltdb shipper in 0 mode"
level=info ts=2021-10-23T18:20:44.273754879Z caller=worker.go:109 msg="Starting querier worker using query-scheduler and scheduler ring for addresses"
level=info ts=2021-10-23T18:20:44.275038576Z caller=mapper.go:46 msg="cleaning up mapped rules directory" path=/loki/rules-temp
level=error ts=2021-10-23T18:20:44.275154931Z caller=mapper.go:50 msg="unable to read rules directory" path=/loki/rules-temp err="open /loki/rules-temp: no such file or directory"
level=info ts=2021-10-23T18:20:44.280571229Z caller=module_service.go:64 msg=initialising module=store
level=info ts=2021-10-23T18:20:44.281341914Z caller=module_service.go:64 msg=initialising module=server
level=info ts=2021-10-23T18:20:44.282154844Z caller=module_service.go:64 msg=initialising module=memberlist-kv
level=info ts=2021-10-23T18:20:44.283122725Z caller=module_service.go:64 msg=initialising module=ring
level=info ts=2021-10-23T18:20:44.283569491Z caller=ring.go:276 msg="ring doesn't exist in KV store yet"
level=info ts=2021-10-23T18:20:44.283727739Z caller=client.go:254 msg="value is nil" key=collectors/ring index=1
level=info ts=2021-10-23T18:20:44.284877595Z caller=module_service.go:64 msg=initialising module=query-scheduler
level=info ts=2021-10-23T18:20:44.285414633Z caller=module_service.go:64 msg=initialising module=ingester
level=info ts=2021-10-23T18:20:44.285723091Z caller=lifecycler.go:531 msg="not loading tokens from file, tokens file path is empty"
level=info ts=2021-10-23T18:20:44.288407605Z caller=lifecycler.go:560 msg="instance not found in ring, adding with no tokens" ring=ingester
level=info ts=2021-10-23T18:20:44.288765828Z caller=lifecycler.go:407 msg="auto-joining cluster after timeout" ring=ingester
level=info ts=2021-10-23T18:20:44.289072258Z caller=module_service.go:64 msg=initialising module=query-frontend-tripperware
level=info ts=2021-10-23T18:20:44.28655247Z caller=module_service.go:64 msg=initialising module=ingester-querier
level=info ts=2021-10-23T18:20:44.289576422Z caller=module_service.go:64 msg=initialising module=compactor
level=info ts=2021-10-23T18:20:44.287502788Z caller=module_service.go:64 msg=initialising module=distributor
level=info ts=2021-10-23T18:20:44.290379945Z caller=module_service.go:64 msg=initialising module=querier
level=info ts=2021-10-23T18:20:44.29051262Z caller=module_service.go:64 msg=initialising module=query-frontend
level=info ts=2021-10-23T18:20:44.290550373Z caller=module_service.go:64 msg=initialising module=ruler
level=info ts=2021-10-23T18:20:44.29066096Z caller=ruler.go:446 msg="ruler up and running"
level=error ts=2021-10-23T18:20:44.291213038Z caller=client.go:154 msg="error getting key" key=schedulers/scheduler err="Get \"http://localhost:8500/v1/kv/schedulers/scheduler\": dial tcp 127.0.0.1:8500: connect: connection refused"
level=info ts=2021-10-23T18:20:44.291607328Z caller=lifecycler.go:531 msg="not loading tokens from file, tokens file path is empty"
level=info ts=2021-10-23T18:20:44.291770326Z caller=lifecycler.go:560 msg="instance not found in ring, adding with no tokens" ring=distributor
level=info ts=2021-10-23T18:20:44.29194457Z caller=lifecycler.go:407 msg="auto-joining cluster after timeout" ring=distributor
level=info ts=2021-10-23T18:20:44.293356794Z caller=compactor.go:261 msg="compacting table" table-name=index_18909
level=info ts=2021-10-23T18:20:44.295330888Z caller=table.go:78 table-name=index_18909 msg="listed files" count=2
level=info ts=2021-10-23T18:20:44.295438834Z caller=table.go:91 table-name=index_18909 msg="skipping compaction since we have just 2 files in storage"
level=info ts=2021-10-23T18:20:44.29569799Z caller=compactor.go:266 msg="finished compacting table" table-name=index_18909
level=info ts=2021-10-23T18:20:44.295770922Z caller=compactor.go:261 msg="compacting table" table-name=index_18910
level=info ts=2021-10-23T18:20:44.296191195Z caller=table.go:78 table-name=index_18910 msg="listed files" count=3
level=info ts=2021-10-23T18:20:44.29624726Z caller=table.go:91 table-name=index_18910 msg="skipping compaction since we have just 3 files in storage"
level=info ts=2021-10-23T18:20:44.296461776Z caller=compactor.go:266 msg="finished compacting table" table-name=index_18910
level=info ts=2021-10-23T18:20:44.296497052Z caller=compactor.go:261 msg="compacting table" table-name=index_18911
level=info ts=2021-10-23T18:20:44.296781742Z caller=table.go:78 table-name=index_18911 msg="listed files" count=3
level=info ts=2021-10-23T18:20:44.296860263Z caller=table.go:91 table-name=index_18911 msg="skipping compaction since we have just 3 files in storage"
level=info ts=2021-10-23T18:20:44.29702663Z caller=compactor.go:266 msg="finished compacting table" table-name=index_18911
level=info ts=2021-10-23T18:20:44.297066368Z caller=compactor.go:261 msg="compacting table" table-name=index_18912
level=info ts=2021-10-23T18:20:44.297311223Z caller=table.go:78 table-name=index_18912 msg="listed files" count=1
level=info ts=2021-10-23T18:20:44.297352492Z caller=table.go:91 table-name=index_18912 msg="skipping compaction since we have just 1 files in storage"
level=info ts=2021-10-23T18:20:44.2974833Z caller=compactor.go:266 msg="finished compacting table" table-name=index_18912
level=info ts=2021-10-23T18:20:44.297523289Z caller=compactor.go:261 msg="compacting table" table-name=index_18913
level=info ts=2021-10-23T18:20:44.297758512Z caller=table.go:78 table-name=index_18913 msg="listed files" count=1
level=info ts=2021-10-23T18:20:44.29779524Z caller=table.go:91 table-name=index_18913 msg="skipping compaction since we have just
version: "3"
services:
  grafana-loki1:
    command:
      - -config.file=/etc/loki/local-config.yaml
    container_name: grafana-loki1
    cpu_shares: 50
    entrypoint:
      - /usr/bin/loki
    environment:
      - PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    hostname: grafana-loki1
    #image: grafana/loki:main-c4562f1
    image: grafana/loki:latest
    ipc: shareable
    labels:
      com.docker.compose.config-hash: 8b82b629f1ff4edfcb2c3bb61ed6a3ff7cc6dc289f12c3ea0b6a7b870b2ac35e
      com.docker.compose.container-number: 1
      com.docker.compose.oneoff: False
      com.docker.compose.project: loki
      com.docker.compose.project.config_files: /volume1/_DEVICES_BACKUP/dockers/containers-compose/grafana-loki1.yaml
      com.docker.compose.project.working_dir: /volume1/_DEVICES_BACKUP/dockers/containers-compose
      com.docker.compose.service: grafana-loki1
      com.docker.compose.version: 1.28.6
#      org.opencontainers.image.created: 2021-10-15T14:46:30Z
      org.opencontainers.image.revision: 096cf0cfaf3fe62cb1bcc86945229e64dfa326ed
      org.opencontainers.image.source: https://github.com/grafana/loki.git
      org.opencontainers.image.url: https://github.com/grafana/loki
    logging:
      driver: db
      options: {}
    mem_limit: 1073741824
    memswap_limit: 2147483648
    networks:
      - npm
    ports:
      - 0.0.0.0:3100:3100/tcp
    restart: always
    stdin_open: true
"/tmp/grafana-loki1.yaml" [readonly] 47L, 1650C                                                                 1,1           Top
version: "3"
1 files in storage"
level=info ts=2021-10-23T18:20:44.297950487Z caller=compactor.go:266 msg="finished compacting table" table-name=index_18913
level=info ts=2021-10-23T18:20:44.297999808Z caller=compactor.go:261 msg="compacting table" table-name=index_18914
level=info ts=2021-10-23T18:20:44.298234756Z caller=table.go:78 table-name=index_18914 msg="listed files" count=2
level=info ts=2021-10-23T18:20:44.29827312Z caller=table.go:91 table-name=index_18914 msg="skipping compaction since we have just 2 files in storage"
level=info ts=2021-10-23T18:20:44.298425205Z caller=compactor.go:266 msg="finished compacting table" table-name=index_18914
level=info ts=2021-10-23T18:20:44.298458143Z caller=compactor.go:261 msg="compacting table" table-name=index_18915
level=info ts=2021-10-23T18:20:44.298671265Z caller=table.go:78 table-name=index_18915 msg="listed files" count=2
level=info ts=2021-10-23T18:20:44.298718169Z caller=table.go:91 table-name=index_18915 msg="skipping compaction since we have just 2 files in storage"
level=info ts=2021-10-23T18:20:44.298862693Z caller=compactor.go:266 msg="finished compacting table" table-name=index_18915
level=info ts=2021-10-23T18:20:44.298903279Z caller=compactor.go:261 msg="compacting table" table-name=index_18916
level=info ts=2021-10-23T18:20:44.299146977Z caller=table.go:78 table-name=index_18916 msg="listed files" count=2
level=info ts=2021-10-23T18:20:44.299185887Z caller=table.go:91 table-name=index_18916 msg="skipping compaction since we have just 2 files in storage"
level=info ts=2021-10-23T18:20:44.299334329Z caller=compactor.go:266 msg="finished compacting table" table-name=index_18916
level=info ts=2021-10-23T18:20:44.299366477Z caller=compactor.go:261 msg="compacting table" table-name=index_18917
level=info ts=2021-10-23T18:20:44.299630872Z caller=table.go:78 table-name=index_18917 msg="listed files" count=3
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xb55eee]

goroutine 375 [running]:
github.com/grafana/dskit/services.(*FailureWatcher).WatchManager.func1({0x2701f00, 0xc000642780})
	/src/loki/vendor/github.com/grafana/dskit/services/failure_watcher.go:33 +0xae
github.com/grafana/dskit/services.funcBasedManagerListener.Failure(...)
	/src/loki/vendor/github.com/grafana/dskit/services/manager.go:321
github.com/grafana/dskit/services.(*Manager).serviceStateChanged.func1({0x26b16b0, 0xc0007fc0c0})
	/src/loki/vendor/github.com/grafana/dskit/services/manager.go:185 +0x32
github.com/grafana/dskit/services.(*Manager).AddListener.func1()
	/src/loki/vendor/github.com/grafana/dskit/services/manager.go:245 +0x57
created by github.com/grafana/dskit/services.(*Manager).AddListener
	/src/loki/vendor/github.com/grafana/dskit/services/manager.go:243 +0x1c6

This is the configuration

auth_enabled: false

server:
  http_listen_port: 3100
  log_level: "info"

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 1h       # Any chunk not receiving new logs in this time will be flushed
  max_chunk_age: 1h           # All chunks will be flushed when they hit this age, default is 1h
  chunk_target_size: 1048576  # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
  chunk_retain_period: 30s    # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
  max_transfer_retries: 0     # Chunk transfers disabled

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/boltdb-shipper-active
    cache_location: /loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks

compactor:
  working_directory: /loki/boltdb-shipper-compactor
  shared_store: filesystem

limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h

#chunk_store_config:
#  max_look_back_period: 336h

table_manager:
  retention_deletes_enabled: true
  retention_period: 336h

ruler:
  storage:
    type: local
    local:
      directory: /loki/rules
  rule_path: /loki/rules-temp
  alertmanager_url: http://localhost:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true