Hello everyone,
I have an installation of grafana and loki on the same VM with filesystem storage.
I have not enabled compactor but logs are deleted after some days.
This moment doing a query from grafana gives results from past 10 days and chunks in filesystem are from past 21 days.
There should exist data from over 3 months.
Loki config is a default one:
auth_enabled: true
server:
http_listen_port: 3100
grpc_listen_port: 9096
log_level: debug
grpc_server_max_concurrent_streams: 1000
common:
instance_addr: 127.0.0.1
path_prefix: /tmp/loki
storage:
filesystem:
chunks_directory: /tmp/loki/chunks
rules_directory: /tmp/loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
ingester_rf1:
enabled: false
query_range:
results_cache:
cache:
embedded_cache:
enabled: true
max_size_mb: 100
schema_config:
configs:
- from: 2020-10-24
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
pattern_ingester:
enabled: true
metric_aggregation:
enabled: true
loki_address: 192.168.100.120:3100
ruler:
alertmanager_url: http://192.168.100.120:9093
frontend:
encoding: protobuf
Loki by default keeps logs forever, if I remember correctly.
Are you using a mounted docker volume for your Loki container? Or are you running it bare metal on your VM?
Hi Tony, thank you for your reply.
Its bare metal on VM.
That’s what i have seen in every document/forum, loki does not delete anything without configuring compactor.
I have read many cases on google, but most of them ask how to limit retention policy.
Interesting.
You wouldn’t happen to have anything on the system level that cleans up filesystem, would you? Check cronjobs and logrotate, for example.
Also, check in your filesystem and see if the older chunk files are actually removed.
There is a log rotate, but how would that affect loki logs?
Logrotate confs that exists, are for firewalld,chrony,dnf etc, but not for loki.
Chunk files exists but are deleted every 21 days. Till yesterday filesystem had chunks of 08/05, today older chunks are from 09/05
I re-read your configuration, I would recommend you to change your chunk directory to something other than /tmp. Most linux clears out /tmp once in a while.
I have multi tenancy enabled. Do you know any separate config file that could affect retention ?
I have enabled audit and i have below logs. I can see DELETE actions by loki “compactor” ??
type=PATH msg=audit(05/31/2025 19:07:28.863:21522) : item=1 name=/tmp/loki/compactor/index_20239/1748706323-grafana-loki-1729696916241963142.tsdb-tmp inode=134217969 dev=fd:00 mode=file,644 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.909:21523) : item=1 name=/tmp/loki/compactor/index_20239/tenant2/1748704648912298997-compactor-1748647856236-1748702486228-71d1d193.tsdb-tmp inode=201326826 dev=fd:00 mode=file,644 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.910:21524) : item=1 name=/tmp/loki/compactor/index_20239/tenant2/1748704648912298997-compactor-1748647856236-1748702486228-71d1d193.tsdb inode=201326827 dev=fd:00 mode=file,644 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.910:21525) : item=1 name=/tmp/loki/compactor/index_20239/1748706323-grafana-loki-1729696916241963142.tsdb inode=134217970 dev=fd:00 mode=file,644 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.928:21528) : item=1 name=/tmp/loki/compactor/index_20239/tenant2/index-5d483ceaf289dbe7.staging_tmp_p inode=201326828 dev=fd:00 mode=file,644 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.930:21529) : item=1 name=/tmp/loki/compactor/index_20239/tenant2/index-5d483ceaf289dbe7.staging_tmp_po inode=201326829 dev=fd:00 mode=file,644 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.931:21530) : item=2 name=/tmp/loki/compactor/index_20239/tenant2/index-5d483ceaf289dbe7.staging inode=201326827 dev=fd:00 mode=file,644 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.936:21531) : item=1 name=/tmp/loki/compactor/index_20239/tenant2/1748707648932021987-compactor-1748647856236-1748704760239-b6bf8d7e.tsdb.temp inode=201326828 dev=fd:00 mode=file,644 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.936:21532) : item=1 name=/tmp/loki/compactor/index_20239/tenant2/1748707648932021987-compactor-1748647856236-1748704760239-b6bf8d7e.tsdb inode=201326827 dev=fd:00 mode=file,644 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.936:21533) : item=1 name=/tmp/loki/chunks/index/index_20239/tenant2/1748704648912298997-compactor-1748647856236-1748702486228-71d1d193.tsdb.gz inode=134217972 dev=fd:00 mode=file,644 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.936:21534) : item=1 name=/tmp/loki/chunks/index/index_20239/1748706323-grafana-loki-1729696916241963142.tsdb.gz inode=268435654 dev=fd:00 mode=file,644 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.936:21535) : item=1 name=/tmp/loki/compactor/index_20239 inode=134217963 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.936:21536) : item=1 name=/tmp/loki/compactor/index_20239 inode=134217963 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.936:21537) : item=1 name=index_20239 inode=134217963 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.936:21538) : item=1 name=tenant1 inode=167772381 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.936:21539) : item=1 name=tenant1 inode=167772381 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.936:21540) : item=1 name=tenant2 inode=201326825 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.936:21541) : item=1 name=tenant2 inode=201326825 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.937:21542) : item=1 name=index_20239 inode=134217963 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.937:21543) : item=1 name=/tmp/loki/compactor/index_20238 inode=234881195 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.937:21544) : item=1 name=/tmp/loki/compactor/index_20238 inode=234881195 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.937:21545) : item=1 name=index_20238 inode=234881195 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.937:21546) : item=1 name=tenant1 inode=268435654 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.937:21547) : item=1 name=tenant1 inode=268435654 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.937:21548) : item=1 name=tenant2 inode=301990177 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.937:21549) : item=1 name=tenant2 inode=301990177 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.937:21550) : item=1 name=index_20238 inode=234881195 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.938:21551) : item=1 name=/tmp/loki/compactor/index_20237 inode=37773 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.938:21552) : item=1 name=/tmp/loki/compactor/index_20237 inode=37773 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.938:21553) : item=1 name=index_20237 inode=37773 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.938:21554) : item=1 name=tenant1 inode=33588647 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.938:21555) : item=1 name=tenant1 inode=33588647 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.938:21556) : item=1 name=tenant2 inode=67309540 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.938:21557) : item=1 name=tenant2 inode=67309540 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.938:21558) : item=1 name=index_20237 inode=37773 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.939:21559) : item=1 name=/tmp/loki/compactor/index_20236 inode=100674537 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(05/31/2025 19:07:28.939:21560) : item=1 name=/tmp/loki/compactor/index_20236 inode=100674537 dev=fd:00 mode=dir,755 ouid=loki ogid=loki rdev=00:00 obj=system_u:object_r:tmp_t:s0 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
It’s normal for compactor to delete index files during compaction.
As mentioned above, you should configure Loki to save logs to somewhere other than /tmp.
