My current set-up: I have an aws ec2 instance for monitoring services which runs dockerized grafana(grafana:8.3.4) and loki(loki:2.5.0). Logs from multiple other services running on other ec2 instances are being sent to this loki instance by dockerized promtail running on those other ec2 instances. Right now I’m using boltdb and filesystem as storage, so the data will be stored inside the container and I’m volume persisting the /loki/data folder inside the container to local filesystem so that I dont lose any data on container restart.
What I’m looking for: Is it possible to rotate the data when I hit disk usage limit on the ec2 instance, for example move the old loki data to a remote storage like AWS S3 and then loki will continue to use filesystem as storage, and in any case that I want to browse the older logs I just copy the older loki data from S3 onto the loki instance filesystem so that I can browse them? If this is not possible is there another way to rotate the loki data for safe consumption of it later.
Is it also possible to push old logs to loki? For example I’ve started the grafana-loki service today. But my services have been running and generating logs for a month. So is it possible to push those older logs with their appropriate timestamps to loki?