Logs deletion problem

Grafana Loki
1

Hi,
I have the following configuration in Loki but the oldest logs are not deleted (now set to 168h), the compaction is done correctly but it does not delete, can you help me find the problem?

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 172.25.144.232
  path_prefix: /opt/loki
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
   - from: "2020-10-24"
     store: boltdb-shipper
     object_store: filesystem
     schema: v11
     index:
       prefix: index_
       period: 24h

storage_config:
  filesystem:
     directory: /opt/loki/chunks

compactor:
  working_directory: /opt/loki/compactor
  delete_request_store: filesystem
  compaction_interval: 5m
  retention_enabled: true
  deletion_mode: filter-and-delete
  retention_delete_delay: 5m

limits_config:
  retention_period: 168h
  max_query_lookback: 168h


ruler:
  alertmanager_url: http://172.25.144.232:9093

Thanks

2

I don’t see anything obviously wrong, any error log from compactor?

1 Like
3

I have configured Loki to use TSDB with the same filesystem storage, since boltdb-shipper is deprecated, with the following configuration, is it correct?:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  ring:
    instance_addr: 172.25.144.232
    kvstore:
      store: inmemory
  replication_factor: 1
  path_prefix: /opt/loki

schema_config:
  configs:
  # Old boltdb-shipper schema
  - from: "2020-10-24"
    store: boltdb-shipper
    object_store: filesystem
    schema: v11
    index:
      prefix: index_
      period: 24h
  # New TSDB schema below
  - from: "2025-01-16"
    store: tsdb
    object_store: filesystem
    schema: v13
    index:
      prefix: index_
      period: 24h


storage_config:
  # New tsdb-shipper configuration
  tsdb_shipper:
    active_index_directory: /opt/loki/tsdb-index
    cache_location: /opt/loki/tsdb-cache

  filesystem:
    directory: /opt/loki/chunks

compactor:
  working_directory: /opt/loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 10m
  retention_delete_worker_count: 150
  delete_request_store: filesystem
  deletion_mode: filter-and-delete

limits_config:
  retention_period: 168h
  max_query_lookback: 168h

but now it is not writing to “tsdb-index” or “/chunks” either, and “tsdb-cache” is not writing excessively either (I don’t know if it’s normal) as you can see below:

drwxr-xr-x 7 loki nogroup 85 Jan 15 15:26 tsdb-index
drwxr-xr-x 4 loki nogroup 5300224 Jan 16 01:32 chunks
drwxr-xr-x 3 loki nogroup 25 Jan 16 07:26 tsdb-cache
drwxr-xr-x 4 loki nogroup 39 Jan 16 10:26 compactor
drwxr-xr-x 4 loki nogroup 92 Jan 16 10:31 wal

Also the index where it is writing is inside the /chunks directory (I don’t know where that configuration comes from. I have set /opt/loki/chunks):

/opt/loki/chunks/index$ ls -lrt
total 0
drwxr-xr-x 2 loki nogroup 32 Jan 14 12:36 delete_requests
drwxr-xr-x 2 loki nogroup 39 Jan 16 00:26 index_20102
drwxr-xr-x 2 loki nogroup 37 Jan 16 01:56 index_20103
drwxr-xr-x 3 loki nogroup 75 Jan 16 10:31 index_20104

I’m using Loki_2.9.2 version

4

Looks correct to me.

You have configured your filesystem directory to /opt/loki/chunks, this means that Loki will use that as the root directory. Chunks will be written to /opt/loki/chunks/<TENANT>, index will be written to /opt/loki/chunks/index.

The active_index_directory is the directory where Loki keeps the index file “before” it’s uploaded to the backend storage. For you it’s all on the same filesystem, but if you were using S3 as storage you’d see active_index_directory on your Loki instance, and the actual index would be on S3/index.

1 Like
5

Ok, so by using filesystem I understand that I don’t need to configure the following

tsdb_shipper:
    active_index_directory: /opt/loki/tsdb-index
    cache_location: /opt/loki/tsdb-cache

But I still don’t understand why it doesn’t save the chunks in the directory, the last ones are from Jan 16 01:3x, about 20 hours ago

/opt/loki$ ls -lrt /opt/loki/chunks/ | tail -n 5
-rw-r–r-- 1 loki nogroup 14133 Jan 16 01:29 ZmFrZS81ZmRjOWMzMDY0OTZlNGU1OjE5NDZjNGNhZjkyOjE5NDZjYmFkMDczOmQ5YjJkN2Ex
-rw-r–r-- 1 loki nogroup 121627 Jan 16 01:31 ZmFrZS9lNTg1NjMxOWJhNzllMzBhOjE5NDZjNGU4NzY0OjE5NDZjYmNhMmM4Ojk3NzNhNjA0
-rw-r–r-- 1 loki nogroup 874927 Jan 16 01:31 ZmFrZS82ODIzMDg2MjMzMzY4ZWMxOjE5NDZjNGU3MTkyOjE5NDZjYmNjNDczOjUzNzgyNjE5
-rw-r–r-- 1 loki nogroup 56560 Jan 16 01:32 ZmFrZS83MDVjNTBkZmNlMTA1Yjc3OjE5NDZjNGVmMDNjOjE5NDZjYmQwODhlOmEzYmY3Y2Y5
drwxr-xr-x 44 loki nogroup 4096 Jan 16 19:47 fake

And other question, Is it recommended that I install the latest version? 3.x

6

I think you misunderstood. You should still configure those, but they are not where the final storage for index is. those are for “active index”, meaning index that haven’t been written to the storage yet.

My understanding is chunk files are written to a directory corresponding to tenant name. If you don’t have multi tenant enabled, then the default tenant name is fake. so I’d check under the fake directory and see if the chunk files are perhaps written there.

No reason not to. I’d of course test it first with your current configuration on a dev node.

1 Like
7

I’ve seen that it saves all the chunks in the fake folder since I don’t have any tenant configured, now I have to confirm that it deletes the logs after the days I have configured, because it seems that it doesn’t delete the chunks that it had before moving to v13 and TSDB. I’ll keep you informed

Even if I don’t configure it, it creates them automatically:

drwxr-xr-x 7 loki nogroup 85 Jan 16 21:58 tsdb-shipper-active
drwxr-xr-x 3 loki nogroup 25 Jan 16 21:59 tsdb-shipper-cache

Thanks!!

8

Did you get this to work? I am also finding the storage/retention config pretty confusing - an inherited 2.8.2 installation here…My retention settings seem to be being ignored. I am running with a simple local filesystem set up.

I assume it is possible to retrospectively clear down indexes and chunks? i.e. this system has been running for months and I would now like to introduce limited retention - I assume this is possible but the alternative might be that data is somehow marked appropriately at ingestion time and retention will work from now on?

And one specific question if anyone has answer - path_prefix in common does not have any explanation in the config reference - is this essentially the base folder for all of the others in the storage or common configs (unless they are absolute paths)?

Furthermore, is tsdb as a storage option purely for indexes? i.e. chunks still need to be defined separately?

Thanks

9

OK, I can see there is a great deal of confusion on various fora about how to get this working, going back literally years - a reasonable summary is here. Like many great open source efforts, it baffles me that people are willing to put in the effort to write amazing, market-leading products and code - but not fully document them!!

I will continue to struggle with the many config settings here until I get a result, but it absolutely does not behave as expected… any suggestions gratefully received.

10

Can you post your actual questions? I’ll see if I can help.

To be honest, I haven’t had too much trouble with Loki’s documentation. Could it be better? Sure, but it’s not the worst and I personally find it serviceable. If you manage to figure things out that you find documentation unclear about perhaps also consider contributing.