grafana 8.3.3 // promtail 2.3.0
I regulary get a promtail error:
promtail-linux-arm[12676]: level=error ts=2022-01-12T09:54:16.087596489Z caller=client.go:355 component=client host=192.168.2.57:3100 msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): entry with timestamp 2022-01-12 09:54:05.981574586 +0000 UTC ignored, reason: 'entry out of order' for stream: {filename=\"/etc/unbound/log/unbound.log\", job=\"unboundlogs\"},"
my promtail config:
server:
http_listen_port: 9080
grpc_listen_port: 0
log_level: error
positions:
filename: /tmp/positions.yaml
clients:
- url: http://192.168.2.57:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: redirectlogs
__path__: /var/log/pihole/redirect.log
- job_name: system2
static_configs:
- targets:
- localhost
labels:
job: unboundlogs
__path__: /etc/unbound/log/unbound.log
the unbound log rotates daily
/etc/unbound/log/unbound.log {
daily
rotate 1
copytruncate
delaycompress
compress
notifempty
missingok
get a lot of errors on the unbound log (very active log), fewer errors on the redirect log (also daily rotates)
anything I can do / configure to eliminate these errors, I sometimes need to restart promtail to get results in the dashboard, using this datasource?