Cannot query old logs from managed loki instance

I am using promtail to push all my historical logs (from 1 week ago) to grafana cloud managed loki instance, using the timestamp stage to set the log’s timestamp to when it was created.

Using grafana cloud explore - I am only able to see logs that were created from 3 hours ago, and all the older logs were not visible/query-able

From promtail log level debug - showed that all logs were pushed successfully to loki

Couple of things to note when pushing old logs:

  1. I’ve not used Grafana Cloud, so I am not sure how you go about configuring it, but first you want to make sure reject_old_samples is false.

  2. When replaying old log, you must make sure the logs are sent in order, from old to new, because you cannot write logs with old timestamp to a log stream with new data already written. Sometimes this is less than obvious, for example if you are parsing every long line for timestamp, and one of the line happens to be malformed and not be included in your parser selector then it could get sent to loki with current timestamp, something to watch out for.