In grafana loki we have 1 metrics
grafana_loki_production.loki_distributor_ingester_append_failures.count
what the meaning of this metrics
is this indicates we loose our logs or there is retry mechanism in it
how could we decrease it
According to the documentation it means:
The total number of failed batch appends sent to ingesters.
See Observability | Grafana Loki documentation. I suppose it represents some sort of issue between your distributors and ingesters, if the value is too high.
i am using simple scalable ingester and distributor are on the same pod.
is there any docs that state how to reduce it because on the log its just says
level=error ts=2024-02-04T18:46:29.310594247Z caller=index_set.go:140 msg="failed to close temp file" path=/var/loki/tsdb-shipper-active/multitenant/my-logs_19757/1707071429-loki-write-1-1704190870918191079.tsdb.temp err="close /var/loki/tsdb-shipper-active/multitenant/my-logs_19757/1707071429-loki-write-1-1704190870918191079.tsdb.temp: file already closed"
what could we do. already increasing ram but not much result
I am running simple scalable mode as well, and I don’t believe I’ve seen that before. I’d recommend you to double check and make sure that connectivity between all your write containers are working for all the ports specified (gossip, grpc, http). Also post your configuration please, and perhaps a screen shot of your /ring output.