Loki logs retention

Grafana Loki
1

I have had an installation of Loki set up for a while now, and I use Grafana as the interface to access the logs. It seems like I can only access logs up to 24h old, so I took a look at my Loki config, where I have the following schema config:

schema_config:
  configs:
    - from: 2018-04-15
      store: boltdb-shipper
      object_store: aws
      schema: v11
      index:
        prefix: loki_index_
        period: 24h

I tried to add an additional config to the list, like so:

    - from: 2021-04-21
      store: boltdb-shipper
      object_store: aws
      schema: v11
      index:
        prefix: loki_index_
        period: 168h

However, when I deploy this config and restart Loki it fails saying that future date schema configurations must have the period set to 24h. Am I going about this the right way? How can I update my schema so I can access logs older than 24h?

Thanks,
Guy

2

No the schema should always be set to 24h, and has nothing to do with retention. There’s 2 dials within loki that you might be getting hit by.
1 - the retention settings, where Loki will just ‘forget’ about the logs (but it doesn’t delete them from S3 yet, the Loki team says that is coming very soon)
2 - lookback settings, which controls the max point where loki will actually perform searches

3

Thanks for the detailed reply, but after further investigation it turned out that my issue isn’t related to Loki at all, but it’s instead due to the max lines display setting in grafana which was set to 1000. When I increased it I was able to see back further.

I’m looking forward to the new pagination features that are being developed for grafana as the current behaviour is kind of fiddly for users.

Anyway, thanks again for your help!

4

Its very easy to see why the Loki team set that limit - you can just keep increasing it and watch in real time as your browser/host tanks under the increased load, especially once you start adding client side transformations. The pagination is already in Grafana on the main branch, and I imagine the 8.0 release will coincide with GrafanaCON so it should be out soon.

1 Like
5

Oh for sure. I tried 5000 and it worked, but it definitely struggles. It’ll do us until 8.0 is out.

6

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.