Loki - Log deletion with line filtering not working


What am I doing
Deleting logs from Loki via compactor API endpoint

What I want to achieve
Delete logs based on stream and line filters

My current results
I can only delete logs based on stream filters, but not line

Loki version: 2.6.1


I have a quite big deployment of Loki with almost all components, for storage I use S3. From time to time, I need to delete some logs that match specific streams and line filters. Whenever I include line filters in a query Loki does not delete any of the logs. I’ll share more details below.


Relevant part of Loki config

  compactor_address: loki-distributed-compactor.loki.svc.cluster.local:3100

  compaction_interval: 10m
  delete_request_cancel_period: 15m
  deletion_mode: filter-and-delete
  retention_delete_delay: 2h
  retention_enabled: true
  shared_store: s3

  chunk_idle_period: 30m
  chunk_retain_period: 1m

  cache_results: true
       enable_fifocache: false
         addresses: dns+loki-distributed-memcached-frontend.loki.svc.cluster.local:11211

     cache_ttl: 24h
     shared_store: s3

Relevant part of Overrides config

     allow_deletes: true

Relevant part of the gateway (Nginx) config

location = /loki/api/v1/delete {
    set $loki_api_v1_delete_backend http://loki-distributed-compactor.loki.svc.cluster.local;
    proxy_pass       $loki_api_v1_delete_backend:3100$request_uri;
    proxy_http_version 1.1;


I have a specific service that I trigger to create logs with the label {facility: "archilogen"}, I want to delete logs only containing the message “archilogen-warn”. I first generate logs, then wait for 30m+ (maybe even multiple hours) so chunks become idle and are sent to S3. After this I send the following API request to the compactor:

curl -I -X POST -H 'X-scope-OrgID: main' -g 'http://xxx/loki/api/v1/delete?query={facility="archilogen"}|="archilogen-warn"'
HTTP/1.1 204 No Content

I can see that request was successful:

curl -H 'X-scope-OrgID: main' 'http://xxx:3020/loki/api/v1/delete' | jq '.[] | select(.status=="received")'

  "request_id": "7cdae6e5",
  "start_time": 0,
  "end_time": 1667381172.093,
  "query": "{facility=\"archilogen\"}|=\"archilogen-warn\"",
  "status": "received",
  "created_at": 1667381172.093

Theoretically according to my config they should be deleted in 15m, Yet after many hours, non of the logs are deleted.

On the other hand, if I issue the following request, I’ll definitely see that there are matching logs

curl -H 'X-scope-OrgID: main' -g 'http://xxx:3020/loki/api/v1/query_range?query={facility="archilogen"}|="archilogen-warn"&start=1667281172' | jq

And what’s even more strange, If I set query= parameter to simply {facility="archilogen"} all logs will be deleted within 15-30m

One thing I suspected initially was caching, but in this case similar would happen when using label filters only.

Would you have any suggestions?