Some of the log lines are not ingested into loki by promtail with write error: msg=“write operation failed” details=“entry too far behind,” even with ‘reject_old_samples: false’ in loki configuration.
Example error line from loki: msg=“write operation failed” details="entry too far behind, entry timestamp is: 2025-02-28T02:38:32Z, oldest acceptable timestamp is: 2025-02-28T06:30:41Z for stream
For the very first file read, promtail is expected to ingest around 29k logs from the log file but only 27k logs are ingested, others are failing with write error: msg=“write operation failed” details=“entry too far behind,”
LOKI Config:
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
log_level: debug
grpc_server_max_concurrent_streams: 1000
grpc_server_max_recv_msg_size: 8388608
grpc_server_max_send_msg_size: 8388608
common:
instance_addr: 127.0.0.1
path_prefix: 'C:\Program Files\GrafanaLabs\loki\tmp\loki'
storage:
filesystem:
chunks_directory: 'C:\Program Files\GrafanaLabs\loki\tmp\loki\chunks'
rules_directory: 'C:\Program Files\GrafanaLabs\loki\tmp\loki\rules'
replication_factor: 1
ring:
kvstore:
store: inmemory
query_range:
results_cache:
cache:
embedded_cache:
enabled: true
max_size_mb: 100
limits_config:
metric_aggregation_enabled: true
max_query_length: 0
retention_period: 86400h
reject_old_samples: false
max_cache_freshness_per_query: 10m
per_stream_rate_limit: 512M
per_stream_rate_limit_burst: 1024M
cardinality_limit: 200000
ingestion_burst_size_mb: 1000
ingestion_rate_mb: 10000
max_entries_limit_per_query: 1000000
max_label_value_length: 20480
max_label_name_length: 10240
max_label_names_per_series: 300
max_query_series: 100000
schema_config:
configs:
- from: 2020-10-24
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
pattern_ingester:
enabled: true
metric_aggregation:
loki_address: 127.0.0.1:3100
ruler:
alertmanager_url: http://127.0.0.1:9093
frontend:
encoding: protobuf
analytics:
reporting_enabled: false
compactor:
working_directory: 'C:\Program Files\GrafanaLabs\loki\tmp\loki\retention'
retention_enabled: true
retention_delete_delay: 2m
delete_request_store: filesystem
PROMTAIL Config:
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: 'C:\Program Files\GrafanaLabs\promtail\tmp\promtail\positions.yaml'
clients:
- url: http://127.0.0.1:3100/loki/api/v1/push
scrape_configs:
- job_name: newregex-3
static_configs:
- targets:
- localhost
labels:
job: newregex-3
__path__: 'D:\misc\mvsn.log'
host: flexnet.otxlab.net
pipeline_stages:
- regex:
expression: '^.[0-9]+:[0-9]+:[0-9]+\s\(mvsn\)\s(?P<action>IN|OUT|DENIED):\s"(?P<feature>[a-zA-Z0-9\.]+)"\s(?P<user>[a-zA-Z0-9_-]+)@(?P<machine>[a-zA-Z0-9_-]+)(\s+|\s+\(.*\))\^\^\^\s(?P<c_version>[0-9]+\.[0-9]+)\s(?P<uname>[a-zA-Z0-9_-]+)\s(?P<ip_addr>[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)\s([0-9]+ 0x[a-zA-Z0-9]+|[0-9]+ 0)\s[0-9]+\s[0-9]+\s(?P<duration>[0-9]+:[0-9]+:[0-9]+)\sN/A\s[0-9]+\s[0-9]+\s(?P<stamp>[0-9]+\-[0-9]+\-[0-9]+ [0-9]+:[0-9]+:[0-9]+).*$'
- template:
source: stamp
template: '{{ regexReplaceAll "(^|\\s|:|\\-)(\\d)(:|\\s|\\-|$)" .Value "${1}0${2}${3}" }}'
- template:
source: stamp
template: '{{ regexReplaceAll "(:|\\-)(\\d)(:|\\-)" .Value "${1}0${2}${3}" }}'
- template:
source: stamp
template: '{{ regexReplaceAll ":(\\d)$" .Value ":0${1}" }}'
- template:
source: stamp
template: '{{ regexReplaceAll "([0-9]+)\\-([0-9]+)\\-([0-9][0-9][0-9][0-9])" .Value "${3}-${2}-${1}" }}'
- timestamp:
source: stamp
format: 2006-01-02 15:04:05
location: 'America/New_York'
- labels:
action:
feature:
user:
machine:
c_version:
ip_addr:
duration:
- drop:
source: action
value: ""
drop_counter_reason: "not_required"
Requesting suggestions/solutions/workarounds