Tempo doesn't take any traces. Nothing in logs

[abellindsey]

Hi
I am trying to run tempo-distributed in kubernetes with minio s3 storage on virtual host is the same cloud.

custom.yaml file to apply

---
storage:
  trace:
    backend: s3
    s3:
      access_key: 'acc_key'
      secret_key: 'secret_key'
      bucket: 'echo-test'
      endpoint: 'endpoint with minio on 9000 port (API)'
      insecure: true
traces:
  otlp:
    grpc:
      enabled: true
    http:
      enabled: true
  zipkin:
    enabled: true
  jaeger:
    thriftHttp:
      enabled: true
  opencensus:
    enabled: true

minio:
  enabled: false

server:
  # -- Log level. Can be set to trace, debug, info (default), warn, error, fatal, panic
  logLevel: debug
distributor:
  config:
    # -- Enable to log every received trace id to help debug ingestion
    # -- WARNING: Deprecated. Use log_received_spans instead.
    log_received_traces: true
    # -- Enable to log every received span to help debug ingestion or calculate span error distributions using the logs
    log_received_spans:
      enabled: true
      include_all_attributes: true
      filter_by_status_error: true

So, after deploy in k8s all pods green and seems everything is ok.
But, when i try to push some traces on distributor nothing happens and no single-tenant blocks appear in s3
Curl post send partial success, but also nothing

[mariorodriguez]

Hi @abellindsey. Are ingested spans being logged in the distributor? Or not even that?

If nothing is logged in the distributor, it’s likely that your exporter is failing to send traces to Tempo.

[abellindsey]

You mean that traces dont reach distributor?
I have an example curl request for OTLP http receiver like that

and it shows partial success
Partial success seems to be a distributor response?

[abellindsey]

Also, logs about accepted traces are not found either in the distributor or in the ingester.

[mariorodriguez]

Tempo uses OTel components in the distributor to receive and parse traces. That response is indicating that some data was rejected by the OTel receiver. Given that you’re sending a single trace, it’s rejecting everything.

Verify that the payload adheres to the OTLP spec. For example, the trace ID is not following the spec.

[abellindsey]

I repeat this test scenario for OTLP with traces. Also partial success. I changed nothing in this example.

Could it be misunderstanding between tempo and OTLP/HTTP spec?

[ismailyenigul]

Hi @mariorodriguez
I have the same issue. deployed grafana agent on kubernetes with helm. trying to send sample trace to grafana cloud with the following command.
trace.json is from opentelemetry-proto/examples/trace.json at main · open-telemetry/opentelemetry-proto · GitHub at March 20, 2024

curl -X POST -H "Content-Type: application/json" -d @trace.json -i grafana-k8s-monitoring-grafana-agent.monitoring.svc.cluster.local:4318/v1/traces

HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 20 Mar 2024 10:48:01 GMT
Content-Length: 21

{"partialSuccess":{}}

I can see the service name in grafana cloud explore section by choosing tempo.
It shows the service name in search filter. But there is no trace id in the search result. it is just empty.

No idea how to debug. I enabled grafana-agent debug logging but I dont see any useful log to understand the problem.

If the json file is invalid, is it an issue with version of OTLP/HTTP used in grafana agent?