Separate storage for logs with specific label

Grafana Loki
1

Hi,
We are running a scalable loki in kubernetes with promtail agents and minio storage.

We have a task of transitioning some of the logs, depending on label, to separate s3 bucket for longer storage.
We have a specific label for them, but can’t find a clean way to do it.

As far as we could find, the option is to use a compactor with a selector, but can’t find a way to use different buckets with it, or set up some kind of granular transitioning for the logs.

limits_config:
  retention_period: 744h
  retention_stream:
  - selector: '{label="CustomLabel"}'
    priority: 1
    period: 24h

If anyone has a way/example how to do it - would be really grateful.

thanks.

2

There isn’t a built-in forwarding feature from Loki, so you’ll have to come up with something, and depending on your expectation and desired outcome the solutions can be different.

One key element you did not mention is what format you want the logs to be in when moving to a separate S3 bucket. Are you looking for those logs to be in plain-text format (human readable), or are you looking for them to be in Loki’s chunk/index format? This will decide what approach you take.

If you are looking to do plain-text long-term storage, the easiest way to do this would probably be some sort of scripted exporter. Loki’s API is quite good and pretty easy to use, the only thing you need to watch out for is Loki has a limit on number of results that can be returned, and I don’t believe there is a way to incorporate that into your API call dynamically (no paging support). So you can’t query for a day of logs, for example, and expect pages of results to return should the number exceed the limit. So you’ll have to do a bit of work by calculating the number of result within a given time frame, and if it’s bigger than your limit, shrink the time frame some what, until you find a time range that works, then loop through the original window and download all the logs. You can then ship those to your long term storage however you see fit.

If you are looking to do another Loki-style storage, the easiest approach would probably to just setup another Loki cluster for longer-term storage, and configure promtail agents to send to two Loki clusters. I don’t believe you can configure two Loki endpoints within one promtail agent, so you may need to have multiple. You might be able to do it with grafana agent, but I am not 100% certain on that.

1 Like
3

Thanks for the answer!

Basically we need them for regulations purposes, so it’s not really important in what format to save them as they won’t be accessed that often. They just have to be kept somewhere.

we’ve thought about this, the problem is - if we need to save some other logs as well. It just doesn’t scale as it would require more and more loki instances(

We’ll move forward with an exporter solution for now.

1 Like