Some (not all) traces are disappearing after 2 days

Grafana Tempo
1

Hi Team,

I am facing an issue where some error traces disappeared after 2 days. I will really appreciate any help here!

Here are the details on my setup & config.
I used following chart to install tempo in the Kubernetes cluster on AWS.

tempo = {
      name             = "tempo"
      repository       = "https://grafana.github.io/helm-charts"
      chart            = "tempo-distributed"
      version          = "1.8.5"
      namespace        = "tempo"
    }

Following is the configuration I am using for the tempo.

serviceAccount:
  name: tempo
  annotations:
    eks.amazonaws.com/role-arn: "${irsa_arn}"
global:
  clusterDomain: "cluster.local"
gateway:
  enabled: true
minio:
  enabled: false
storage:
  trace:
    backend: s3
    s3:
      bucket: "${bucket_name}"
      endpoint: "s3.${bucket_region}.amazonaws.com"
      insecure: false
      region: "${bucket_region}"
traces:
  otlp:
    http:
      enabled: true
    grpc:
      enabled: true
distributor:
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: "0.0.0.0:4317"
          http:
            endpoint: "0.0.0.0:4318"