No service graph data found Tempo

[maximilianofilloycba]

Hello, I have the following error.

I also see that the following happens to me

it keeps loading and never shows me the results

Conf:

Tomcat – port 4317 → otel --port 4017–> Tempo – Port 3200 → Grafana

Tomcat conf:

        -javaagent:/opt/tempo/opentelemetry-javaagent.jar
        -Dotel.traces.exporter=otlp
        -Dotel.exporter.otlp.endpoint=http://10.250.2.89:4317
        -Dotel.metrics.exporter=none
        -Dotel.logs.exporter=none
        -Dotel.exporter.otlp.protocol=grpc
        -Dotel.service.name=ehcos04
        -Dotel.resource.attributes=application=ehcos04

Otel Conf:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:55681

exporters:
  logging:
    loglevel: debug
  otlp/tempo:
    endpoint: "10.250.2.89:4017"
    tls:
      insecure: true
  prometheus:
    endpoint: "0.0.0.0:8889"

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging, otlp/tempo]
    metrics:
      receivers: [otlp]
      exporters: [logging, prometheus]
    logs:
      receivers: [otlp]
      exporters: [logging]

In the “otel” logs I see that data is arriving from tomcat

Otel logs:

otel-collector-1  | Span #335
otel-collector-1  |     Trace ID       : ae6f30492d3b80c2c13ba15e63179196
otel-collector-1  |     Parent ID      : 3177c70f6d39c586
otel-collector-1  |     ID             : 6364d5ab28235838
otel-collector-1  |     Name           : Transaction.commit
otel-collector-1  |     Kind           : Internal
otel-collector-1  |     Start time     : 2024-05-28 19:55:31.517557901 +0000 UTC
otel-collector-1  |     End time       : 2024-05-28 19:55:31.518230479 +0000 UTC
otel-collector-1  |     Status code    : Unset
otel-collector-1  |     Status message :
otel-collector-1  | Attributes:
otel-collector-1  |      -> thread.id: Int(224)
otel-collector-1  |      -> thread.name: Str(Clinic Scheduler_Worker-7)

Tempo Conf:

stream_over_http_enabled: true
server:
  http_listen_port: 3200
  log_level: info
  grpc_server_max_recv_msg_size: 1.572864e+07
  grpc_server_max_send_msg_size: 1.572864e+07

query_frontend:
  search:
    duration_slo: 5s
    throughput_bytes_slo: 1.073741824e+09
  trace_by_id:
    duration_slo: 5s

distributor:
  receivers:                           # this configuration will listen on all ports and protocols that tempo is capable of.
    jaeger:                            # the receives all come from the OpenTelemetry collector.  more configuration information can
      protocols:                       # be found there: https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver
        thrift_http:                   #
        grpc:                          # for a production deployment you should only enable the receivers you need!
        thrift_binary:
        thrift_compact:
    zipkin:
    otlp:
      protocols:
        http:
        grpc:
    opencensus:

ingester:
  max_block_duration: 5m               # cut the headblock when this much time passes. this is being set for demo purposes and should probably be left alone normally

compactor:
  compaction:
    block_retention: 1h                # overall Tempo trace retention. set for demo purposes

metrics_generator:
  registry:
    external_labels:
      source: tempo
      cluster: docker-compose
  storage:
    path: /var/tempo/generator/wal
    remote_write:
      - url: http://10.250.2.89:9090/prometheus/api/v1/write
        send_exemplars: true
  traces_storage:
    path: /var/tempo/generator/traces

storage:
  trace:
    backend: local                     # backend configuration to use
    wal:
      path: /var/tempo/wal             # where to store the wal locally
    local:
      path: /var/tempo/blocks

overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics, local-blocks] # enables metrics generator
    global:
      max_bytes_per_trace: 50000000

Logs Tempo:

tempo-1  | level=warn ts=2024-05-28T19:57:43.561953057Z caller=pool.go:250 msg="removing distributor_metrics_generator_pool failing healthcheck" addr=127.0.0.1:9095 reason="failing healthcheck status: NOT_SERVING"
tempo-1  | level=warn ts=2024-05-28T19:57:43.562233573Z caller=pool.go:250 msg="removing distributor_pool failing healthcheck" addr=127.0.0.1:9095 reason="failing healthcheck status: NOT_SERVING"
tempo-1  | level=info ts=2024-05-28T19:57:48.997102332Z caller=registry.go:232 tenant=single-tenant msg="collecting metrics" active_series=804
tempo-1  | level=warn ts=2024-05-28T19:57:58.561657195Z caller=pool.go:250 msg="removing distributor_metrics_generator_pool failing healthcheck" addr=127.0.0.1:9095 reason="failing healthcheck status: NOT_SERVING"
tempo-1  | level=warn ts=2024-05-28T19:57:58.56179246Z caller=pool.go:250 msg="removing distributor_pool failing healthcheck" addr=127.0.0.1:9095 reason="failing healthcheck status: NOT_SERVING"
tempo-1  | level=info ts=2024-05-28T19:58:03.997819565Z caller=registry.go:232 tenant=single-tenant msg="collecting metrics" active_series=804
tempo-1  | level=warn ts=2024-05-28T19:58:13.561861786Z caller=pool.go:250 msg="removing distributor_pool failing healthcheck" addr=127.0.0.1:9095 reason="failing healthcheck status: NOT_SERVING"
tempo-1  | level=warn ts=2024-05-28T19:58:13.562018324Z caller=pool.go:250 msg="removing distributor_metrics_generator_pool failing healthcheck" addr=127.0.0.1:9095 reason="failing healthcheck status: NOT_SERVING"
tempo-1  | level=info ts=2024-05-28T19:58:18.995888045Z caller=registry.go:232 tenant=single-tenant msg="collecting metrics" active_series=804
tempo-1  | level=warn ts=2024-05-28T19:58:28.562238676Z caller=pool.go:250 msg="removing distributor_metrics_generator_pool failing healthcheck" addr=127.0.0.1:9095 reason="failing healthcheck status: NOT_SERVING"
tempo-1  | level=warn ts=2024-05-28T19:58:28.562483142Z caller=pool.go:250 msg="removing distributor_pool failing healthcheck" addr=127.0.0.1:9095 reason="failing healthcheck status: NOT_SERVING"