We’re going to use tsdb-shipper and minio, and the index and chunks stored in minio will be backed up daily using a separate backup solution and kept for a maximum of 2 years. We’ll use compactor and limit_config to set the retention period to 2 months. I understand that data is deleted after 2 months, but in case of any issues later, is there a way to restore data older than 2 months from the backup system and reload it into Loki so we can view it again?
You can restore index files pretty easily, chunk files however are more random. If you need to restore something from backup you’ll need to:
- First determine the index files to restore, and restore them.
- Parse the index files for list of all chunks the indexes refer to. I don’t believe there is a convenient script that does this, so you may have to write your own GO script for this.
- Restore the chunks.