Hey Folks,
I have configured a log retention policy in Loki to store logs for 30days, but someway somehow my logs are being kept for only 10 days.
For instance, today’s date is 2025-06-27 and based on my configured log retention policy, I should be able to query logs as far back as 2025-06-01, but I can only get logs up to 2025-06-17 which seems to me the logs in loki are only retained for 10days.
What is the issue here and how can I fix this?
Reference doc: Log retention | Grafana Loki documentation
This is my loki retention configuration:
limits_config:
metric_aggregation_enabled: true
ingestion_rate_mb: 10
ingestion_burst_size_mb: 20
ingestion_rate_strategy: global
retention_period: 720h
compactor:
working_directory: /tmp/loki/compactor
compaction_interval: 10m
retention_enabled: true
retention_delete_delay: 2h
retention_delete_worker_count: 150
delete_request_store: filesystem
schema_config:
configs:
- from: 2020-10-24
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
This is my entire configuration
auth_enabled: false
server:
http_listen_port: <port>
grpc_listen_port: <port>
log_level: info
grpc_server_max_concurrent_streams: 1000
common:
instance_addr: <localhost>
path_prefix: /tmp/loki
storage:
filesystem:
chunks_directory: /tmp/loki/chunks
rules_directory: /tmp/loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
query_range:
results_cache:
cache:
embedded_cache:
enabled: true
max_size_mb: 100
limits_config:
metric_aggregation_enabled: true
ingestion_rate_mb: 10
ingestion_burst_size_mb: 20
ingestion_rate_strategy: global
retention_period: 720h
compactor:
working_directory: /tmp/loki/compactor
compaction_interval: 10m
retention_enabled: true
retention_delete_delay: 2h
retention_delete_worker_count: 150
delete_request_store: filesystem
schema_config:
configs:
- from: 2020-10-24
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
pattern_ingester:
enabled: true
metric_aggregation:
loki_address: localhost:<port>
ruler:
alertmanager_url: http://localhost:<port>
frontend:
encoding: protobuf
# querier:
# engine:
# enable_multi_variant_queries: true
# Optional: disable anonymous reporting
# analytics:
# reporting_enabled: false
In my compactor, I see some delete items in the directory, the content in the “deletion” file is binary so i’m not sure if the deletion is actually happening
[root@ip-10-0-0-41 compactor]# du -sh *
0 delete_requests
16K deletion
0 retention
And to answer your third question, I do not have any local filesystem clean up process
I would advise you first to change your chunks and rules directories to something other than /tmp
.
I have updated the chunk and rules directory to /etc/loki
and also changed the retention policy to 3 days so that it can be easily verified if there have been any change. I will keep this thread updated of the results after 5 days (on Friday).
This is my current configuration:
auth_enabled: false
server:
http_listen_port: <port>
grpc_listen_port: <port>
log_level: info
grpc_server_max_concurrent_streams: 1000
common:
instance_addr: <localhost>
path_prefix: /etc/loki
storage:
filesystem:
chunks_directory: /etc/loki/chunks
rules_directory: /etc/loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
query_range:
results_cache:
cache:
embedded_cache:
enabled: true
max_size_mb: 100
limits_config:
metric_aggregation_enabled: true
ingestion_rate_mb: 10
ingestion_burst_size_mb: 20
ingestion_rate_strategy: global
retention_period: 72h
compactor:
working_directory: /etc/loki/compactor
compaction_interval: 10m
retention_enabled: true
retention_delete_delay: 2h
retention_delete_worker_count: 150
delete_request_store: filesystem
schema_config:
configs:
- from: 2020-10-24
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
pattern_ingester:
enabled: true
metric_aggregation:
loki_address: localhost:<port>
ruler:
alertmanager_url: http://localhost:<port>
frontend:
encoding: protobuf
# querier:
# engine:
# enable_multi_variant_queries: true
# Optional: disable anonymous reporting
# analytics:
# reporting_enabled: false