Hi All,
I have recently onboarded promtail for log export via Loki,
promtail is throwing below errors while sending the logs to loki
level=error ts=2021-08-03T08:54:29.108262315Z caller=client.go:355 component=client host=prometheus.qa.abc.info:3100 msg=“final error sending batch” status=400 error=“server returned HTTP status 400 Bad Request (400): entry with timestamp 2021-08-03 08:54:28.045712756 +0000 UTC ignored, reason: ‘entry out of order’ for stream: {component=“MongoS”, filename=”/var/log/mongodb/x_mongos/mongodb.log", host=“eu-west-1a-mongos-01”},"
level=error ts=2021-08-03T08:54:30.394298123Z caller=client.go:355 component=client host=prometheus.qa.abc.info:3100 msg=“final error sending batch” status=400 error=“server returned HTTP status 400 Bad Request (400): entry with timestamp 2021-08-03 08:54:29.298853134 +0000 UTC ignored, reason: ‘entry out of order’ for stream: {component=“MongoS”, filename=”/var/log/mongodb/x_mongos/mongodb.log", host=“eu-west-1a-mongos-00”},"
level=error ts=2021-08-03T08:54:31.589429757Z caller=client.go:355 component=client host=prometheus.qa.abc.info:3100 msg=“final error sending batch” status=400 error=“server returned HTTP status 400 Bad Request (400): entry with timestamp 2021-08-03 08:54:30.551714686 +0000 UTC ignored, reason: ‘entry out of order’ for stream: {component=“MongoS”, filename=”/var/log/mongodb/x_mongos/mongodb.log", host=“eu-west-1a-mongos-01”},"
level=error ts=2021-08-03T08:54:32.89161879Z caller=client.go:355 component=client host=prometheus.qa.abc.info:3100 msg=“final error sending batch” status=400 error=“server returned HTTP status 400 Bad Request (400): entry with timestamp 2021-08-03 08:54:31.806009793 +0000 UTC ignored, reason: ‘entry out of order’ for stream: {component=“MongoS”, filename=”/var/log/mongodb/x_mongos/mongodb.log", host=“MongoS-2”},"
level=error ts=2021-08-03T08:54:36.090443113Z caller=client.go:355 component=client host=prometheus.qa.abc.info:3100 msg=“final error sending batch” status=400 error=“server returned HTTP status 400 Bad Request (400): entry with timestamp 2021-08-03 08:54:33.060058347 +0000 UTC ignored, reason: ‘entry out of order’ for stream: {component=“MongoS”, filename=”/var/log/mongodb/x_mongos/mongodb.log", host=“MongoS-2”},"
level=error ts=2021-08-03T08:54:37.197457106Z caller=client.go:355 component=client host=prometheus.qa.abc.info:3100 msg=“final error sending batch” status=400 error=“server returned HTTP status 400 Bad Request (400): entry with timestamp 2021-08-03 08:54:34.313817414 +0000 UTC ignored, reason: ‘entry out of order’ for stream: {component=“MongoS”, filename=”/var/log/mongodb/x_mongos/mongodb.log", host=“MongoS-1”},"
level=error ts=2021-08-03T08:54:38.391665969Z caller=client.go:355 component=client host=prometheus.qa.abc.info:3100 msg=“final error sending batch” status=400 error=“server returned HTTP status 400 Bad Request (400): entry with timestamp 2021-08-03 08:54:37.32240859 +0000 UTC ignored, reason: ‘entry out of order’ for stream: {component=“MongoS”, filename=”/var/log/mongodb/x_mongos/mongodb.log", host=“eu-west-1a-mongos-01”},"
Could you send me any reference to fix these errs.
Thanks in advance,
Regards,
Naveen