Loki indexing not showing logs older than 18 hours

Setup on K8s - grafana/loki helm chart, storage on s3 with multiple replicas for write, read and backend. Grafana is using gateway as datasource. Version used app-version 2.9.1 and chart version 5.23.1. Retention set to 5d

Problem I have - Everything works and queries are fast, however the log indexing show logs under only last 18 hours or so and nothing older than a day, s3 bucket contains logs for the last few days but loki doesn’t show that. Checked via logcli as well and see the same issue with error `No logs volume available for current queries and time range”.
If I move my storage to local storage with persistent volume, I don’t see this issue. 2.9.1 readme was supposed to fix missing historical indexes so is anyone else seeing this

Helm chart values gist:bf9de9b49860f1448ed2627c3e628d4f · GitHub

1 Like