Loki-compactor active_retention metrics are 0 and missing

Grafana Loki
1

Describe the bug
The loki-compactor active_retention metrics are 0 and some of them are missing related to namespace: loki_cpmpactor, but we confirmed that the deletions actually happen in the S3 bucket.

To Reproduce
Steps to reproduce the behavior:

  1. Started Loki 3.1.0
  2. scrape metrics from loki compactor service
  3. found that metrics are missing
  4. example: loki_compactor_apply_retention_last_successful_run_timestamp_seconds 0, loki_compactor_apply_retention_operation_total no-data

Expected behavior
Retention metric should be available because retention is happening

Environment:

  • Infrastructure: [Kubernetes]
  • Deployment tool: [helm]

Screenshots, Promtail config, or terminal output
Configuration that we used for the compactor:

rentention_period: 183d #this is for Global and for some tenants we apply 14d
compactor:
      compaction_interval: 10m
      delete_request_cancel_period: 24h
      retention_enabled: true
      retention_delete_delay: 2h
      delete_request_store: s3
2

Not seeing this problem. I just checked and I see values for both of those metrics, running Loki 3.2.1. Couple of questions and things to try:

  1. Are you running simple scalable mode?
  2. How many compactors do you have?
  3. If you just hit the /metrics endpoint on your compactor, are you not seeing those metrics still? If you have multiple compactors be sure to do it on all of them.
3

Thanks @tonyswumac.
I try things that asked.

  1. Are you running simple scalable mode? We are using Loki Distributed
  2. How many compactors do you have? 1 Compactor
  3. If you just hit the /metrics endpoint on your compactor, are you not seeing those metrics still? If you have multiple compactors be sure to do it on all of them. 1 Compactor only running and still missing

We are using the same configuration deployment on another environment, and it is working fine there. However, we are not able to find with the provided configuration why these metrics are not coming here.

4

Interesting. I am not sure what’s wrong. One more thing I’d try, since you mentioned that you have another cluster that’s functioning properly, perhaps try hitting the /config endpoint on compactors from both cluster, then do a diff and see what is different.

5

Hi @tonyswumac . i didn’t find any difference in the config.
I have a question: can data volume impact no retention metrics? retention is happening but maybe not correctly happening?

6

I am not sure, but if it’s not working properly you should be seeing some sort of error logs.