Tempo memory requirement recommendations

Grafana Tempo
1

Hi

I’m running Tempo 2.3.1 and Grafana 10.3.1. We’ve got a relatively small environment and so wondering what the minimum requirements to size up the various tempo components are. We have:

Our set up is:
Grafana, Tempo (micro-service mode with 2 grafanas, 1 compactor, 1 distributor, 3 ingesters, 1 memchached, 1 metrics generator, 1 querier, 1 query front-end, 2 prometheus, 3 loki

Also, general guidance as to whether what we’re seeing makes sense.

  1. We have roughly 500-1000 spans received (tempo_distributor_spans_received_total)
  2. Around 150 traces received.
  3. The Otel Distributor Spans/second shows roughly 600-800 spans/s (tempo_receiver_accepted_spans)
  4. We’re seeing OOMs quite a bit on our ingesters and on the compactor.
  5. You can see our Go heap for compactor and ingesters run out of memory reasonably frequently. (e.g. the sudden increase which then keels over)

Any advice on right-sizing? and any other metrics we can look at to identify what is causing the sudden memory increase and the follow on OOM, rather than just a punt in the dark…

image
image1892×401 158 KB

image
image1905×608 59.4 KB

image
image1738×414 146 KB

2

Is anyone able to share some insights on this? In particular on how to reduce memory consumption on the various ingest/compator side of things for tempo?

3

we have a section on docs that has few tuning guides

without looking at the tempo config it’s kinda hard to give recommendations on tuning :slight_smile: