New TSDB schema causes Loki to fail

Grafana Loki
1

I wanted to enable the new tsdb index store. So I created a new schema. Notice that at the time of running this configuration, the from date is in the future.

        schema_config:
          configs:
            - from: "2022-01-01"
              store: boltdb
              object_store: filesystem
              schema: v11
              index:
                prefix: index_
                period: 168h
#            - from: "2023-06-20"
#              store: tsdb
#              object_store: gcs
#              schema: v12
#              index:
#                prefix: index_
#                period: 24h # 168h

        storage_config:
          boltdb:
            directory: /index

          filesystem:
            directory: /chunks

          gcs:
            bucket_name: monitoring-001-loki

          tsdb_shipper:
            active_index_directory: /tsdb-index
            shared_store: gcs

When I restart Loki with the updated config, Loki fails with the following errors in the logs:

level=info ts=2023-06-16T21:29:44.90262055Z caller=config.go:26 type=config msg="    client_ca_file: \"\""
mkdir : no such file or directory
error initialising module: store
github.com/grafana/dskit/modules.(*Manager).initModule
    /drone/src/vendor/github.com/grafana/dskit/modules/modules.go:122
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
    /drone/src/vendor/github.com/grafana/dskit/modules/modules.go:92
github.com/grafana/loki/pkg/loki.(*Loki).Run
    /drone/src/pkg/loki/loki.go:457
main.main
    /drone/src/cmd/loki/main.go:110
runtime.main
    /usr/local/go/src/runtime/proc.go:250
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1598
level=info ts=2023-06-16T21:29:44.902626501Z caller=config.go:26 type=config msg="    client_auth_type: \"\""
level=info ts=2023-06-16T21:29:44.902673491Z caller=config.go:26 type=config msg="    key_file: \"\""
level=info ts=2023-06-16T21:29:44.902682208Z caller=config.go:26 type=config msg="    cert_file: \"\""
level=info ts=2023-06-16T21:29:44.902688299Z caller=config.go:26 type=config msg="  grpc_tls_config:"
level=info ts=2023-06-16T21:29:44.902694611Z caller=config.go:26 type=config msg="    client_ca_file: \"\""
level=info ts=2023-06-16T21:29:44.902701113Z caller=config.go:26 type=config msg="    client_auth_type: \"\""
level=info ts=2023-06-16T21:29:44.902707425Z caller=config.go:26 type=config msg="    key_file: \"\""
level=info ts=2023-06-16T21:29:44.902714159Z caller=config.go:26 type=config msg="    cert_file: \"\""
level=info ts=2023-06-16T21:29:44.902723186Z caller=config.go:26 type=config msg="  http_tls_config:"
level=info ts=2023-06-16T21:29:44.9027305Z caller=config.go:26 type=config msg="  tls_min_version: \"\""
level=info ts=2023-06-16T21:29:44.902737292Z caller=config.go:26 type=config msg="  tls_cipher_suites: \"\""
level=info ts=2023-06-16T21:29:44.902743604Z caller=config.go:26 type=config msg="  grpc_listen_conn_limit: 0"
level=info ts=2023-06-16T21:29:44.90275165Z caller=config.go:26 type=config msg="  grpc_listen_port: 9095"
level=info ts=2023-06-16T21:29:44.902758433Z caller=config.go:26 type=config msg="  grpc_listen_address: \"\""
level=info ts=2023-06-16T21:29:44.902765226Z caller=config.go:26 type=config msg="  grpc_listen_network: tcp"
level=info ts=2023-06-16T21:29:44.902772249Z caller=config.go:26 type=config msg="  http_listen_conn_limit: 0"
level=info ts=2023-06-16T21:29:44.902780424Z caller=config.go:26 type=config msg="  http_listen_port: 64310"
level=info ts=2023-06-16T21:29:44.902787148Z caller=config.go:26 type=config msg="  http_listen_address: \"\""
level=info ts=2023-06-16T21:29:44.902794021Z caller=config.go:26 type=config msg="  http_listen_network: tcp"
level=info ts=2023-06-16T21:29:44.902800623Z caller=config.go:26 type=config msg=server:
level=info ts=2023-06-16T21:29:44.902806534Z caller=config.go:26 type=config msg="use_sync_logger: true"
level=info ts=2023-06-16T21:29:44.902813167Z caller=config.go:26 type=config msg="use_buffered_logger: true"
level=info ts=2023-06-16T21:29:44.90281959Z caller=config.go:26 type=config msg="ballast_bytes: 0"
level=info ts=2023-06-16T21:29:44.902827304Z caller=config.go:26 type=config msg="http_prefix: \"\""
level=info ts=2023-06-16T21:29:44.902833546Z caller=config.go:26 type=config msg="target: all"
level=info ts=2023-06-16T21:29:44.905362722Z caller=main.go:108 msg="Starting Loki" version="(version=2.8.2, branch=HEAD, revision=9f809eda7)"
level=info ts=2023-06-16T21:29:44.906510729Z caller=server.go:323 http=[::]:64310 grpc=[::]:9095 msg="server listening on addresses"
level=warn ts=2023-06-16T21:29:44.919514606Z caller=cache.go:114 msg="fifocache config is deprecated. use embedded-cache instead"
level=warn ts=2023-06-16T21:29:44.919563168Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - chunksembedded-cache"
level=info ts=2023-06-16T21:29:44.923446754Z caller=table_manager.go:134 msg="uploading tables"
level=error ts=2023-06-16T21:29:44.923608753Z caller=log.go:171 msg="error running loki" err="mkdir : no such file or directory\nerror initialising module: store\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/drone/src/vendor/github.com/grafana/dskit/modules/modules.go:122\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/drone/src/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/drone/src/pkg/loki/loki.go:457\nmain.main\n\t/drone/src/cmd/loki/main.go:110\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598"
2

This seems to be related to an issue on GitHub: Error: “at least one bucket name must be specified”

3

I finally got it working for me. I do not completely understand what is required, but apparently the storage related configurations are very finicky. If you miss a shared_store parameter in one of the modules or mix up the common.storage settings, the system will fail. The error messages are NOT helpful.