Getting "entry too far behind" even if reject_old_samples and reject_old_samples_max_age should allow

Grafana Loki
1

I’m trying to send logs from fluentbit to loki.

My loki configuration is quite simple:

auth_enabled: false

limits_config:
  reject_old_samples: false
  reject_old_samples_max_age: 52w

server:
  http_listen_port: 3100

common:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory
  replication_factor: 1
  path_prefix: /tmp/loki

schema_config:
  configs:
  - from: 2020-05-15
    store: tsdb
    object_store: filesystem
    schema: v13
    index:
      prefix: index_
      period: 24h

storage_config:
  filesystem:
    directory: /tmp/loki/chunks

But I get the following error messages:

rpc error: code = Code(400) desc = entry with timestamp 2025-02-21 05:00:09 +0000 UTC ignored, reason: ‘entry too far behind, entry timestamp is: 2025-02-21T05:00:09Z, oldest acceptable timestamp is: 2025-03-18T05:02:21Z’,
entry with timestamp 2025-02-21 05:00:09 +0000 UTC ignored, reason: ‘entry too far behind, entry timestamp is: 2025-02-21T05:00:09Z, oldest acceptable timestamp is: 2025-03-18T05:02:21Z’,
entry with timestamp 2025-02-22 05:00:02 +0000 UTC ignored, reason: ‘entry too far behind, entry timestamp is: 2025-02-22T05:00:02Z, oldest acceptable timestamp is: 2025-03-18T05:02:21Z’,
entry with timestamp 2025-02-22 05:00:02 +0000 UTC ignored, reason: ‘entry too far behind, entry timestamp is: 2025-02-22T05:00:02Z, oldest acceptable timestamp is: 2025-03-18T05:02:21Z’,
entry with timestamp 2025-02-23 05:00:01 +0000 UTC ignored, reason: ‘entry too far behind, entry timestamp is: 2025-02-23T05:00:01Z, oldest acceptable timestamp is: 2025-03-18T05:02:21Z’,
…
user ‘fake’, total ignored: 51 out of 52 for stream: {job="resticprofile", service_name="resticprofile"}"

Since today’s date is 2025-03-19, I don’t get why the entry is being rejected.
Furthermore, I don’t get why the oldest acceptable timestamp is 2025-03-18…

May someone enlighten me?

2

It’s a tehnology (Loki) limitation: you can’t ingest too old and also too new (that’s your case, you are ingesting logs, which can’t exist now, because their timestamp is > now).
I guess validation doesn’t expect that someone can ingest future logs, so error message is not correct for this edge case.

3

I totally understand your point, but I don’t think that’s the current situation.

As I understand it, the current situation is the following (in time order of the timestamps):

timestamp description
2024-03-xx Timestamp set by the reject_old_samples_max_age option (here 52w, which can be read as now() - '52w')
2025-02-23 05:00:01 +0000 UTC Timestamp of the entry being rejected
2025-03-18T05:02:21Z Oldest acceptable timestamp (as per log)
2025-03-19T10:39:23.609539525Z Timestamp of the ingestion tentative

From what I can tell, the ts of the pushed is neither too far in the past, nor in the future.

To me, it looks like the reject_old_samples_max_age is not taken into account when calculating the 'oldest acceptable timestamp.

4

When you already have logs in a log stream reject_old_samples_max_age is irrelevant.

When you try to write older logs into a log stream (log stream defined as logs with the same set of labels) that already has newer data then it’s considered out-of-oder writes. Loki has a limit of (max_chunk_age/2) when it comes to out-of-order write, and with default 2h of max chunk age that comes down to 1 hour. Essentially, if you have logs in a log stream, the oldest data you can write to the same stream is something 1 hours older than the latest log.

See Grafana Loki configuration parameters | Grafana Loki documentation