I would like to set up retention for logs stored in Grafana Loki and I am not sure how the retention really works. There is more period time to configure and from documentation, it is not so clear to understand how whole process of retention works.
First log at 20:43 and last one ad 23:00. This is quite long according to my estimation. I do not want to use it for this short period of time but how can I estimate log retention for example for 3 days, 1 week or 2 months?
How is retention calculated and processed?
Do I understand Loki configuration parameters properly?
How does retention differs from chunk.period and index.period under schema_config.config?
Why there is no answer to this question? The documentation is ugly, as usual with Grafana, plenty of parameters and no example:
# Tables older than this retention period are deleted. Must be either 0
# (disabled) or a multiple of 24h. When enabled, be aware this setting is
# destructive to data!
# CLI flag: -table-manager.retention-period
[retention_period: <duration> | default = 0s]
Thank you. How about the duration, is it in seconds, minutes, hours, days… F****
Compactor runs in loops every compaction_interval, from docs:
The Compactor loops to apply compaction and retention at every compaction_interval, or as soon as possible if running behind.
It will mark chunks that need to be deleted - this will not delete the chunks!
Retention period - configures when the data will be deleted since compactor marked chunks for deletion. From dcos:
Note that retention is only available if the index period is 24h.
The minimum retention period is 24h.
This means, retention must be longer than 24h. From my experience, it works somehow even on shorter periods, but this is not Loki is designed for.
Response on my questions from this Topic:
1. How is retention calculated and processed?
Compaction runs in a loop (every compaction_interval) or as soon as possible if running behind). Retention also runs in loops and check if chunks are older than retention_period. Keep in mind, all of above runs in loops, here is an example of compacting (I configured compaction_interval to 10m):
From comment I would expect only hours is supported with integer multiplication. This should be enough, I cannot imagine requirements of retention in seconds or minutes. I can see default value is in seconds default = 0s but I would ignore it. Maybe you can try to set duration in hours between 1 and 2 days, e.g. 25h or 32h.
Valid values: 24h, 48h, …, 336h (this is my retention). You can try e.g. 36h and see, if it works.