How do we know how many logs is gone from Loki metrics?

In here Observability | Grafana Loki documentation it mentions loki_distributor_ingester_append_failures_total as “The total number of failed batch appends sent to ingesters.”

But it still left the following question open for me:

  1. What is the loki_distributor_ingester_append_failures_total really is? What happen if the batch from Distributor failed to sent to Ingester? Does it mean the logs is gone? Or it will be retried?
  2. If Log is can probably gone after received by the Distributor but never accepted by the Ingester, how can I know how many logs are gone?
  3. How can I know if Ingester successfully persist all logs into object storage? Also, how can I know how many logs or streams that already received by Ingester but failed to persist?

Thanks in advance.

1 Like