How to gather all logs for annual reports

Grafana Loki
1

I want to do annual reports so I need a way to export all logs at once. I use Loki and I saw that in data source you can export logs. The problem is that it’s limited to 5000 logs which is really not much.

Is there a way to query all existing logs in Loki? Through LogCLI?

2

Don’t know of a way to do this out of the box. Best to write your own script. Fortunately the API is pretty easy to use, so it shouldn’t be too bad.

Your general logic would be something like:

  1. Get start date and end date.
  2. Divide the duration by a certain interval (we used 24h). The reason for this is simple, you don’t want to query for 360 days of data at once, because you’ll either hit a limit or kill your Loki cluster.
  3. Perform API call, get results, save in a text file with a timestamp.
  4. Repeat until done.
3

LogCLI should let you do that. From the docs


You can download an unlimited number of logs in parallel, there are a few flags which control this behaviour:

  --parallel-duration
  --parallel-max-workers
  --part-path-prefix
  --overwrite-completed-parts
  --merge-parts
  --keep-parts

Depending on what sort of reporting you want to do, you might be able to use metrics queries to summarise things. A whole year of data is probably a bit much to try to graph, but per week or per month summaries might work.

4

I am able to download those logs to .part files, but I’m searching for a way to combine them to a single .csv file.
Is there a way to do this using logCLI or do I need my own script?

There is also a --merge-parts option with logCLI but it seems like it outputs the .parts to the console instead of merging them to a single file?

For info, I used the query from the example in the documentation
logcli query --timezone=UTC --from="2023-03-01T10:00:00Z" --to="2023-07-26T20:00:00Z" --output=raw --parallel-duration="15m" --parallel-max-workers="4" --part-path-prefix="/tmp/jobname" --keep-parts --merge-parts '{job="jobname"}'