Cannot access existing schema after TSDB migration

Grafana Loki
1

Hi, I’ve just upgraded from 2.9.2 to 3.4.2 via helm and migrated to TSDB. The new TSDB v13 schema is working fine, but the backend seems unable to query anything in the existing schema. The schema config looks like this:

  schemaConfig:
    configs:
      - from: "2022-01-11"
        index:
          period: 24h
          prefix: loki_index_
        object_store: gcs
        schema: v12
        store: boltdb-shipper
      - from: "2024-04-02"
        index:
          period: 24h
          prefix: loki_index_
        object_store: gcs
        schema: v13
        store: tsdb

When trying to access data from the existing schema, the error is encountered:

index set  has some problem, cleaning it up" err="invalid tsdb path: /var/loki/tsdb-shipper-cache/loki_index_20177/compactor-1743431358.r
failed to init user index set" err="invalid tsdb path: /var/loki/tsdb-shipper-cache/loki_index_20177/compactor-1743431358.r"

/var/loki/tsdb-shipper-cache is a valid directory on the pod with data in it. It seems that the compacted data files for the new schema are now in a different naming format and are now within a tenant subdirectory in the storage bucket.

Any ideas as to why it can’t access the old schema?
Thanks

2

What errors are you getting from queriers?

3

Hi Tony - no errors from the queriers, they just don’t return any data from that range.

4

It was my mistake, I had the wrong start date in the TSDB config, so it was trying to access old data with the TSDB schema rather than BoltDB schema. :face_with_peeking_eye: