Hello,
We are in the process of migrating from Jaeger UI with Elasticsearch storage to Tempo with S3 storage. Some of our older services rely on fetching trace information from Jaeger UI for analysis. These services typically operate at a maximum of 20 queries per second (qps), with queries spanning a relatively short time range.
During this transition, we noticed that Tempo does not offer a batch API to load trace information, forcing us to send requests individually for each trace ID. Unfortunately, this method frequently triggers an error response: “There are too many awesome requests.”
Is there a way to batch retrieve trace information in Tempo? Additionally, we have observed some interesting metrics that might be relevant, which can be found here: Tempo Querier High Request Volume and 206 Response Code.
4.15 update: Now we increased the number of Tempo querier pods to 32, but this adjustment seems to have had minimal effect. Additionally, it appears that only one block is being primarily queried at any given time; could there be a rate limit in S3 that we are encountering?
Thank you for your assistance.