Different retention period per stream using Azure Blob

Grafana Loki
1

Hi,
We would like to implement different retention periods for operational logs and audit logs using Azure Blob storage

Setting per stream retention period works the way that data are not searchable, but they stay in storage, and the only possibility of deleting data is using lifecycle management.

I have also tried setting up multi-tenant environment in the hope that it will separate data to different folders but they stay still in fake folder.

Is there any possibility of how to do it, or will I need to set up two different Lokis with different storage and lifecycle management?

thanks for the response.

2

How did you determine that the data are still in the storage?

This is definitely a misconfiguration.

In general, I think either solution you already attempted would do what you need, and if you found them to not work then I think it’s most likely some configuration issue that can be fixed. If you have the freedom to separate audit logs from regular logs via tenant ID, that’s probably the easiest way to go. I’d say evaluate and pick a solution you think will work best for you, if it doesn’t work share your configuration and we can try and figure it out.

3

I have this in the loki config:

common:
  instance_addr: 127.0.0.1
  path_prefix: /opt/loki
  storage:
    azure:
      account_name: XXX
      container_name: loki-prod
      use_managed_identity: true
      endpoint_suffix: blob.core.windows.net
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

compactor:
  working_directory: /opt/loki/retention
  compaction_interval: 1m
  retention_enabled: true
  retention_delete_delay: 1m
  retention_delete_worker_count: 150
  delete_request_store: azure
  delete_request_cancel_period: 10m

querier:
  max_concurrent: 4

limits_config:
  max_streams_per_user: 0
  max_global_streams_per_user: 0
  ingestion_rate_mb: 24
  metric_aggregation_enabled: true
  retention_period: 90d

and storage usage is just increasing

image
image904×718 42.4 KB

When I had 90d lifecycle management enabled, data were deleted. When I changed it back to 365d and kept a 90d retention period, it stopped dropping down.

Hmm, it seems that it works.

image
image610×342 6.28 KB

But it seems that there is some delay until it is moved to the “tenant” folder. When it is day one, it is kept in the index folder, and after the compactor kicks in, it moves it to the tenant folder. Is that a correct assumption?

4

There is no delay. Loki doesn’t move existing data into tenant folder, it only does it for data that come in after tenant is enabled.

5

I get it now. They are stored in the cache on the server and pushed to blob storage afterward. My bad.

what about that growing storage even if retention is enabled?

6

It’s hard to tell from your screenshot exactly what’s happening.