Overrides: local rate strategy vs. global rate strategy

Grafana Tempo
1

Hey community

I have deployed tempo-distributed, and I have been seeing the distributor component dropping spans occasionally. It seems it may hit some limits regarding trace size. In this diagram, I can see some significant numbers of spans being discarded under the reason trace_too_large, which tells us the traces are exceeding the size limit set by the max_byte_size_per_trace threshold.

image
image1892×745 149 KB

The max_byte_size_per_trace and max_traces_per_user values were increased, and this helped to reduce the spans dropped by the distributor instance. However, discards are still happening. The current configuration I have is a local rate strategy for ingesting traces like this:

overrides:
  defaults:
    global:
      max_bytes_per_trace: 7000000
    ingestion:
      max_traces_per_user: 30000
    metrics_generator:
      processors:
      - service-graphs
      - span-metrics
  per_tenant_override_config: /runtime-config/overrides.yaml

I got motivated to share this situation here to double-check with you my understanding of how the ingestion limits are applied in tempo regarding the overriding strategy and how the use of a global one can help avoid spans being dropped.
So regarding local strategy, is it okay to consider that when a distributor instance reaches its ingestion limit, the load is not shared with other instances? That is why I am seeing spans dropped.
I guess, as the documentation implies, distributors operate independently with no awareness of their peers (others distributor’s instances)

I am thinking of switching to a global rate strategy for managing ingestion limits as it seems to share the load of the data being ingested, allowing for adjustment of the total capacity, with the global limit distributed across the instances my cluster may have.
Then as apparently I don’t see anything else on the documentation, my main question is:
Is the global rate strategy the only alternative I may have to tackle this issue, or are there other ingestion settings I may consider?

By the way, if I work with the max_byte_per_trace at overrides context, will that apply either for ingester, compactor, and queried, too? as it seems to be in this defined variable structure

I would appreciate your thoughts :slightly_smiling_face: