We have set up a small Tempo cluster (three nodes in scalable-single-binary mode), and it’s running mostly fine.
However, we are getting compactor failures almost every compaction cycle):
googleapi: Error 404: No such object: /single-tenant/<block_id>/meta.json
The error message is sometimes slightly different:
-
failed to mark block compacted during retention
-
unable to mark block compacted
-
failed to clear compacted block during retention
but the inner error is always the above
Based on what we can find about these errors, it would appear that the compactor ring is not formed.
However, the ring status page looks fine (all three nodes show as active).
We use Consul as a kv store, not Grafana’s memberlist and GCS as block storage, otherwise our setup should be fairly default.
Any help is appreciated