Promtail: old logs

Grafana Loki
1

Whenever promtails starts first time on a server it goes crazy in attempt to send everything it found to Loki.

The fun fact is that Loki by default wont accept old logs and respond with an error.

The pain here is that while it happens promtail eats CPU resource quite noticeably.

Did not found any workarounds for this, I was expected to something like, either:

  • configuration option like in Kafka topic - aka read log from beginning or from the end in case if there is no stored position
  • configuration option to drop old log lines the same way as it is configured in Loki

Wondering if there are some well known workarounds

2

In this article it explained. But for my also it is not clear. From one side we can see that LOKI accepts old logs only when 1h behind. But on the othe side following configuration is set by default:

reject_old_samples: true
reject_old_samples_max_age: 1w

3

Not exactly, as mentioned, Loki is already rejecting old logs - it is completely fine

The problem is that now, I’m deploying Promtail for a very first time on server

and it starts to process months of logs as crazy eating CPU just to receive an 400 logs are too old from Loki :man_shrugging:

Which is kinda weird

looking for a way to say to promtail - ignore old logs and do not even bother to preprocess and send them to Loki

4

It may be less obvious, but you can use the drop stage in promtail to drop logs older than specified time period. See drop | Grafana Loki documentation.

1 Like
5

In the current version of Loki, 3.3.1 as of this writing, do not forget this bit, as the default value is 2 hours:

# Configure the ingester and how the ingester will register itself to a key value store.
ingester:
  # The maximum duration of a timeseries chunk in memory. If a timeseries runs for
  # longer than this, the current chunk will be flushed to the store and a new
  # chunk created. Default is 2h.
  max_chunk_age: "8766h"

Once old logs are imported, changing this configuration option should not be required anymore.

6

What about the warning from the original article?

The default here is two hours. I strongly discourage increasing this value to create a larger out-of-order window. At Grafana Labs we do not run Loki this way so I can’t easily tell you what challenges you may face. I recommend trying to find a way to use labels to separate the streams so they can be ingested separately.

I am currently struggling with extending the time window for old logs. I would rather not touch this setting - in our environment, the situation where we need to ingest the logs from (for example) last 10 days may happen pretty often, so i would like to set it once and for good. I tried to set reject_old_samples_max_age to 3w and 5000h (while reject_old_samples is set to true), but the Loki still reports:

level=warn ts=2025-01-07T14:33:51.108061786Z caller=grpc_logging.go:76 method=/logproto.Pusher/Push duration=6.108444ms msg=gRPC err="rpc error: code = Code(400) desc = entry with timestamp 2024-12-19 16:10:27.142 +0000 UTC ignored, reason: 'entry too far behind, entry timestamp is: 2024-12-19T16:10:27Z, oldest acceptable timestamp is: 2025-01-07T13:30:25Z'

In other words, Loki still does not accept anything older than 1 hour…

Any word of advise?

7

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.