Can Loki be used to perform data aggregation from a log file?

Grafana Configuration
1
  • What Grafana version and what operating system are you using?

I’m running the latest version of Grafana Docker image; the host machine is running Windows 11.

  • What are you trying to achieve?

I have a Grafana-Loki-Promtail setup that is monitoring a log file. The log file documents the number of files being processed by an application over a period of time. Over the course of a day there could be a hundred data points, e.g., “at 6:00am: 17 files downloaded, 6:01am: 80 files downloaded…”.

I would like to change that so instead of seeing the number of files downloaded at every data point, Loki will consolidate the data hourly, e.g., at 6:00-7:00am, 200 files downloaded, 7:00-8:00am, 600 files downloaded…".

  • How are you trying to achieve it?

I don’t know, that’s why I’m creating this topic. I couldn’t figure it out from the Loki documentation.

  • What happened?

Nothing happened, and that’s the problem.

  • What did you expect to happen?

See above

  • Can you copy/paste the configuration(s) that you are having problems with?

There’s no configuration because I don’t know where to begin

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

No

  • Did you follow any online instructions? If so, what is the URL?

No

2

I would do the aggregation afterwards.
First create a table with Loki (timestamp, value) and aggregate it as a second step.
That’s how I would do it.

Jo

3

Hi, thank you for answering!

I already managed to get the basic setup running (Promtail reads the logs, sends it to Loki, which sends it to Grafana to be rendered into a graph). Now I just need guidance to do the second step (aggregating the data).

Any suggestions you can give about this would be greatly appreciated.

4

What does your tabular data look like? Do you have an example?

5

grafana
grafana1142×217 36.3 KB

In the example above, I want to combine all the data so that the X-axis would be something like 04:00 and the Y-axis will be the sum of file_size_total in all data points from 04:00 - 04:59.