Loki/Promtail : parsing timestamp that are too old

[rossm2]

I tried to ingest old logs into LOKI

scrape_configs:
- job_name: system
  # journal:
  #     from: 3600h 
  pipeline_stages:
    # 2021-01-20T14:01:29: ABC
    - regex:
        expression: '^(?P<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}): .*'
    - timestamp:
        # Mon Jan 2 15:04:05 -0700 MST 2006
        format: '2006-01-02T15:04:05'
        source: time
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

I keep getting this error. too old timestamp … How can I process old logs into loki?

promtail_1 | level=error ts=2021-01-20T20:06:26.4438654Z caller=client.go:305 component=client host=loki:3100 msg=“final error sending batch” status=400 error=“server returned HTTP status 400 Bad Request (400): entry for stream ‘{filename="/var/log/AAAAAA.log", job="varlogs"}’ has timestamp too old: 2020-12-19 12:54:30 +0000 UTC”

Thank you for your time

[wlargou]

Hi,

Which versions are you using, there have been a field added max_age.

Please check this :

promtail: add "max_age" field to configure cutoff for journal reading

Hope it helps.

Good Luck

[rossm2]

I taught max_age was for journal parsing which is not a single log file like in my case.

I took the latest docker image with docker-compose

[rossm2]

the solution was in loki config file. just don’t reject samples

limits_config:
reject_old_samples: false