Loki rotate data tsdb_shipper + filesystem

[starcraftgg97]

Hello,
Could You please tell me how to rotate my Loki data?

Here is my storage config, if anything else is needed, please let me known to share. Thanks!

schema_config:
  configs:
  - from: "2023-01-01"
    index:
      period: 24h
      prefix: index_
    object_store: filesystem
    schema: v12
    store: tsdb

storage_config:
  tsdb_shipper:
    active_index_directory: /var/lib/loki/data/tsdb-index
    cache_location: /var/lib/loki/data/tsdb-cache
    shared_store: filesystem

  filesystem:
    directory: /var/lib/loki/data/

[tonyswumac]

Please see Retention | Grafana Loki documentation.

[starcraftgg97]

hey @tonyswumac,

Thank You for your answer!
Is there anyway to do it manually?

For instance lets say I want to keep 1 year of logs but just 2 months of log in access for Loki. other 10 months should be in a cold storage, and in case I need them, I must be able to restore them.

[tonyswumac]

I don’t personally have a usecase for this, so my comments below are just my thoughts without actual implementations. That side, I think you can approach this in one of two ways:

1. Manual export from Loki to text-based log files with timestamps. Essentially identify the log streams you deem important enough to keep for longer retention, write some sort of script to manually export those to a separate cold storage.

This is probably a better approach if you only need to keep part of your Loki logs.

2. If you are using object storage such as S3 you can configure lifecycle policies to move old index and chunk files to cold storage, and configure your Loki cluster unlimited retention period. The problem with this is difficulty when restoring. The index files can be located by date, and you’d have to parse it to figure out which chunk files to restore from cold storage.

[starcraftgg97]

Thanks!