I have a couple of general questions for which I could not find an answers.
I have a kubernetes cluster and I want to collect the logs from the apps running on it. I have installed promtail and loki (as a single binary, not distributred) on the cluster and promtail is sending logs to loki and loki sends them to S3 on AWS.
And I have configured loki and grafana on my home pc to look the logs from the S3.
When the cluster looses connection to the S3 I saw that the logs are buffered because when I recovered the connection after 15-20min the logs from this interval were sent to the S3. But during this time the memory used by loki increased from 50mb to 100mb.
- Are the chunks or indexes stored in the memory during time when the internet connection is missing?
- Can these buffered chunks and indexes be stored on the hard disk? Because in my case there can be days without internet connection and I don’t want to loose logs nor loki to use a lot of memory ( I want to keep set a limit of loki around 150mb for example and not to fail with out of memory, every buffering to happens on the local storage). Now I removed the connection from about an hour and a half and loki now is using 500mb and that number continues to go up…
- Is my setup ok from architecture point of view? I saw that it’s recommended to use distributed loki on central machine and send logs there directly there from promtail but makes the things a lot more complicated for me.
I am expecting max 1GB logs per day.
This is my Loki configuration:
- from: 2020-10-24
- from: 2021-02-20
cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space
# aws credentials