In here Observability | Grafana Loki documentation it mentions
loki_distributor_ingester_append_failures_total as “The total number of failed batch appends sent to ingesters.”
But it still left the following question open for me:
- What is the
loki_distributor_ingester_append_failures_totalreally is? What happen if the batch from Distributor failed to sent to Ingester? Does it mean the logs is gone? Or it will be retried?
- If Log is can probably gone after received by the Distributor but never accepted by the Ingester, how can I know how many logs are gone?
- How can I know if Ingester successfully persist all logs into object storage? Also, how can I know how many logs or streams that already received by Ingester but failed to persist?
Thanks in advance.