Error when sending logs to loki

Getting this error:

[2022/06/21 21:17:10] [error] [output:loki:loki.0] loki-stack:3100, HTTP status=400 Not retrying.
entry with timestamp 2022-06-21 20:32:01.696331203 +0000 UTC ignored, reason: 'entry too far behind' for stream: {controller_revision_hash="f787b5f7d", k8s_app="kube-proxy", pod_template_generation="6", updateTime="1653791485"},
total ignored: 1 out of 1

Running EKS v 1.22
Installed Loki using helm along with the following values:

loki:
  enabled: true
  persistence:
    enable: true
    size: 50Gi
    storageClassName: efs

configured the fluent bit to collect all the logs. The configuration is the following:

using the following configuration:
    [SERVICE]
        Daemon Off
        Flush {{ .Values.flush }}
        Log_Level {{ .Values.logLevel }}
        Parsers_File parsers.conf
        Parsers_File custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port {{ .Values.metricsPort }}
        Health_Check On
        storage.path              /var/log/flb-storage/
        storage.sync              normal
        storage.checksum          off
        storage.backlog.mem_limit 64M

    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 5M
        storage.type filesystem
        Skip_Long_Lines On

    [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Mem_Buf_Limit 512M
        storage.type filesystem
        Read_From_Tail On

    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

    [OUTPUT]
        name loki
        match *
        host loki-stack
        remove_keys kubernetes,stream
        auto_kubernetes_labels on
        storage.total_limit_size 512M
        log_level {{ .Values.logLevel }}

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.