Logs disappearing after 11 days in storage

Grafana Loki
1

logs delivered by promtail to loki looks deleted in 11 days - query doesn’t find them any more
here’s my /etc/loki/config.yml

auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
log_level: error
grpc_server_max_concurrent_streams: 1000
common:
instance_addr: 172.20.0.239
path_prefix: /tmp/loki
storage:
filesystem:
chunks_directory: /tmp/loki/chunks
rules_directory: /tmp/loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
ingester_rf1:
enabled: false
query_range:
results_cache:
cache:
embedded_cache:
enabled: true
max_size_mb: 100
compactor:
working_directory: /tmp/loki/retention
compaction_interval: 15m
retention_enabled: false
retention_delete_delay: 1h
retention_delete_worker_count: 50
delete_request_store: filesystem
schema_config:
configs:
- from: 2020-10-24
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
limits_config:
ingestion_rate_mb: 4
ingestion_burst_size_mb: 6
reject_old_samples: true
reject_old_samples_max_age: 3w
retention_period: 672h
pattern_ingester:
enabled: true
metric_aggregation:
enabled: true
loki_address: localhost:3100
ruler:
alertmanager_url: http://localhost:9093
frontend:
encoding: protobuf

what am i doing wrong? :frowning:

2

Don’t see anything obvious. Couple of things to try:

  1. Are you sure logs are deleted? check your filesystem for Loki storage and see if there are files older than 11 days.
  2. Are you running Docker? Is the storage mounted properly from host?
  3. Do you have any sort of cronjob that’s doing cleanup outside of Loki for your filesystem?
  4. Do you see logs from compactor that files were deleted?
3

thank you for answer. let’s see…

  1. you’re right! in storage i’ve found files 17-days old. with data. but explore in grafana doesn’t show it. directory /tmp/loki/chunks/fake contains 909054 directories. everyday near 12:00 like 11:56-11:57 i see thousands of directories created in one minute and they all are empty…
  2. no docker. no
  3. no cronjobs. at least i didn’t make any. /etc/crontab is empty
  4. i don’t know how to find compactor logs :frowning:
4

help still needed. anyone some ideas?

5

i’ve disabled compactor like “retention_enabled: false” but nothing changed - explore still cannot find data older than 11 days :frowning: