Total count of logs (not over time)

Grafana Loki
,
1

I see many examples of how to see a count of logs over time, however what I am looking for is how to see counts of logs for all logs.

I see what looks like ways to do this with logcli, but nothing I can use in a dashboard.

1 Like
2

Do count_over_time, with interval of what you want to see (24h, for example, or just $__interval), then set the query type to instant, with exactly 1 data point.

3

I want to see the total count of logs going back as far as data exists (ie; what hasn’t been pruned by retention yet).

count_over_time will only give me a count of the last 24 hours of received logs, will it not? otherwise why would it need a time range?

4

If you want it to go as far back as you can, change it to 365d or something like that. Or use $__interval and change internal on your dashboard.

1 Like
5

Have to tried to use statistics from /metrics
They have values like:
# HELP loki_internal_log_messages_total Total number of log messages created by Loki itself.
# TYPE loki_internal_log_messages_total counter
loki_internal_log_messages_total{level=“debug”} 5.865921e+06
loki_internal_log_messages_total{level=“error”} 289
loki_internal_log_messages_total{level=“info”} 964182
loki_internal_log_messages_total{level=“warn”} 3

curl http://localhost:3100/metrics
Assuming your loki is at port 3100

6

To view a count of all logs in a dashboard, you can use a query like count_over_time({job="your_job"}[1h]) in Grafana if you’re using Loki for log aggregation. This will display the count of logs over time. For all logs, you might want to use a more general query without specific filters, like {job="your_job"}. Then, apply a count function to show the total log count. You can set this up as a panel in your Grafana dashboard for a continuous overview.

7

That is going to run into limits and require a high bump in limits in order to look back that far, so it would appear there is no way to do this via queries that doesn’t require potentially dangerous limit changes.

That would require jacking the limits on the Loki instance waaaaaay up in order to not be blocked, which could cause other problems.

8

This only gives the overall total. I need to be able to total by a filter in order to understand how much of a particular label has consumed my total storage.

9

You cannot do a count function directly on a logs query.

1 Like
10

I think this is for the logs of Loki itself as per the metrics naming.

11

There is such kind of API for querying the chunks and indexes stored in object storage such as S3, which will return the MBs based on tenant and bucket. However, I am not sure whether it supports querying based on period. In our case, we have retention_policy set for each tenant, say 30d or 744h, so we don’t have to worry about the data elder than this period as data has been pruned/deleted.

Besides, as I can recall there is a metrics named loki_distributor_bytes_received_total for the same purpose, which is in counter type. However, this has been removed from the latest doc, you may refer to the elder version such as 2.9.x as below:

12

Quite true, but I don’t know how big your cluster is and whether that is a concern for you or not. Several options I can think of:

  1. Send loki metrics to something like Mimir where long-term querying is not quite as taxing.
  2. Use recoding rule to do the same calculation for a shorter period (30m or 1h), send to mimir, and aggregate there.
  3. Use some sort of script, do the same calculation for a shorter period, send to some sort of database (or you can even send back to Loki) for easier aggregation.
13

Loki doesn’t provide functionality for returning the per-label storage usage. But such a functionality is available in VictoriaLogs - see this question.