Hi, manager here. I am new to Loki and would like to ask about a use case scenario for my teams of developers.
Currently, my teams have been using Prometheus + Grafana to evaluate over time the speed of functions in our web services, count how many times each function is being called, count how many of each HTTP Response codes occurred, etc. Our apps simply take care of all these statistics internally and have Prometheus scrape them from time to time. Then, Grafana comes in for visualization. Things work fine.
But then I heard about Loki just now, and another team (not mine) has just began using it. I see that they have successfully bringing in the logs to Loki and Grafana. I see they could also got all the data such as time taken, what HTTP Response is generated for each request, all other information related to this request, etc. I see they also got alert when particular errors have occurred in the logs.
For the above scenario, I see that Prometheus already scraped all the data and have them in its time series database. Grafana could simply fetch all these numbers, plot them and perform alerts. But for logs, I see that Loki got an engine to manage all the strings. One advantage I see is that you could write anything to log files, and then filter them all out later. But wouldn’t this mean string manipulation and evaluation on the logs all the time? I see the other team just began using it, but I have no information on how will things go in the long run. Right now, I can see things are lightning fast on their Loki + Grafana, but in the long run with longer and longer logs generated daily, will things become slower?
Between what my teams have been doing (Prometheus + Grafana) and Loki, for my Use Case, which is more preferable? Are there significant advantages in the long run that we should switch from Prometheus + Grafana to Loki + Grafana in our Use Case?
How long does Loki keep the logs? Can we specify Loki to keep logs for x months, then move and compress older logs?