Log ingestion size

Hi all,

I’m relatively new to Loki and I’m experimenting with Grafana Cloud free tier. I just received a notification that I have exceeded my monthly log ingestion limit, which is quite weird since I have the grafana agent sending logs only from my local raspberry pi and a small NAS and some probes.

I’ve been trying to find out the cause of this huge log increase over the past week but I can’t seem to figure out the right query and the math doesn’t check out.

All logs should have a label instance that identifies who sent the log, so my query is this:

sum by(instance) (bytes_over_time({instance=~".+"} [$__range])) / 1024^3

The result that I get is a table with the aggregated results. Each line has a different instance name and the column Value #A should show the number of gigabytes sent over the specified range which in this case is 30days. However, as you can see from the picture below, the sum of all of them barely gets to 5gigabytes.

Here is a screenshot from the grafana dashboard related to usage. As you can see it looks like it just wend exponential in the last week.

What am I doing wrong?

Update:

I tried the same query proposed in this tweet and I confirm that it doesn’t look like I ingested more than 5 GigaBytes of logs.

All my logs have the “instance” label, so I’m pretty sure I’m not missing other logs with this query.