We have an installation that previously used boltdb-shipper with local filesystem as a storage. We upgraded it to a new version that now uses boltdb-sipper that sends logs to AWS S3.
I would like to make older logs available for Grafana queries. The new installation works as expected the logs are stored and we can query logs that are stored in S3.
To attach the old data I have configured data schema that refers both old and new storage. Loki does not seem to complain but when I query old data range the results are always empty. The schema configuration was made after the migration so I would accept that some logs of hours during transition may not be available.
Some random inspection of chunks and index segments do shows that they still contain the necessary data. I experimented with a sample of the old data (chunks and indexes) copied to another host and launched loki with the same data schema config to read it. In this case the queries do return old data without issues. So I am 100% confident that the old data is usable.
Is this possible to to have boltdb-shipper to read the data both in local filesystem and remote storage? If so did I miss something in this configuration? Here I have 2 ranges for data schema for old and new data (I excluded config parts that do not seem to be relevant)
schema_config:
configs:
- from: "2022-01-01"
index:
period: 24h
prefix: index_
object_store: filesystem
schema: v11
store: boltdb-shipper
- from: "2023-03-22"
index:
period: 24h
prefix: index_
object_store: s3
schema: v12
store: boltdb-shipper
storage_config:
aws:
bucketnames: a-lovely-bucket
s3: s3://lovely-aws-region
boltdb_shipper:
active_index_directory: /data/loki/boltdb-shipper-active
cache_location: /data/loki/boltdb-shipper-cache
cache_ttl: 24h
filesystem:
directory: /data/loki/chunks
table_manager:
retention_deletes_enabled: false
On another take, if the above is not possible maybe there are other setups, like having 2 separate loki instances that are both configured as data sources in Grafana?