We have configured Tempo with service graphs enabled, and we see service graph nodes and connections for some of our services, but with nodes missing. So a trace that goes between services A->B->C shows as A-C in the service graph.
One thing that looks fishy is that some of the span-ids look like 0000000000000001 or 0000000000000002 although we never see these span ids in HTTP headers.
Not sure these are connected, but any insights are appreciated.
Hi @chrissteinbach.
servicegraph relies on span IDs being unique. If your IDs look like those you share, it’s very possible that the processor is mixing different connections.
What instrumentation library are you using? ID generation is solved in most (all?) common libraries.
We’re using spring boot with org.springframework.cloud:spring-cloud-starter-sleuth. The application designated ‘A’ sends using WebClient, created via a bean. And ‘B’ sends using RestTemplate, also created via a bean.
Tempo shows child spans for A and B but I don’t see any explicit span creation in the code, so not sure where they come from.
Ah, I now understand what’s happening. Spring Boot Sleuth uses zipkin format by default, which Tempo has a few difficulties working with.
Zipkin uses the same ID for parent and children spans, which Tempo does not support—it expects span IDs to be unique across all spans within a trace. That is why you were seeing span IDs of the type 0000000000000001 and 0000000000000002, and why servicegraphs is not working properly. There isn’t really a solution to make zipkin format work natively with Tempo, only workarounds such as the deduping of span IDs we’ve just seen.
I have very little experience with Java, but from what I’ve read, Spring Boot Sleuth supports OTel as tracer. I think using that one you won’t have issues.
This is helpful thank you! There seems to be some experimental support for open telemetry and I’ll look into that after the weekend.
This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.