Hi,
I tried to find any useful information about system requirements about Loki depending of number of logs, especially about storage.
Do you have some docs or threads where is explained how much space using logs and indexes ?
I want to set up Loki and I want to aggregate logs from about 60-70 VPS servers, I need to know how much resources I need it.
Hope someone can help me with that.
Thanks you.
Hi @petarkozic ,
I’m working on the same thing but i don’t have any good number to share yet. But here is what I have found out so far.
To start with, I’m running Loki in Amazon EKS, using S3 as storage. I use the official Grafana loki-distributed helm chart to deploy everything so far.
For my setup it I would first have to know the size of my plain text logs to be able to predict Loki storage requirements.
Once you know that, you can make some calculations based on which chunk_encoding method you use.
The Best Practices documentation has some more info.
Depending on the compression used (we have been using snappy which has less compressibility but faster performance), you need 5-10x or 7.5-10MB of raw log data to fill a 1.5MB chunk.
Ingestion seems to be very performant and when I was using flog for load testing I had no problems ingesting 7500 Apache logs per second with only one ingester node with minimal CPU and RAM.
The tricky part for me so far is to get the data out of Loki in an effective way. So I’m working on optimizing stream count and chunk size and scaling up queriers.
Over all I have been very impressed by Loki so far.
Thank you very much on answer, I hope here is someone who already using Loki in production and want to share this information with us 
We are all on the same journey
To get more accurate answers for you specifically, it would be great if you could share how you are running Loki and what storage type you are using.
For now, I’m running Loki only in docker environment for testing purpose. I want to test how everything works because I’m using Prometheus and Loki is very similar.
I tried to aggregate different type of logs, like weblog, audit, mysql, syslog etc… and I’m very happy how all that works.
I want to make production log server with Loki and I ask this on forum because I want to calculate how much resources I need it. I will use local filesystem as storage, maybe some NFS share.
I have about 60-70 servers which need to aggregate logs into Loki, usually that is web servers.
This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.