Dashboard with Arithmetic Operation Between Two Spans of a Trace

[rajmehta53]

Hey everyone.

We are working on a distributed tracing POC. Our decision to go with Grafana will primarily rely on the possibility of one specific use case. I have attached a visual of an example trace. What we want to calculate is the difference in duration between two spans of the same trace. “SUB” in the visual signifies a subtraction operation.

The scenario for an API call is: We want to calculate the latency that we add to an API call. This means subtracting the time taken by an external client. We want the results as a time-series and create alerts on it. (This basically means that if “OUR” latency crosses a certain threshold, we’ll act on it. Spike in external client latency does not matter.)

Any help on this will be great. Thank you!

EDIT:
I tried doing the following in the Grafana sandbox. Making two queries and then trying to subtract them in the Transform tab. Doesn’t work.

[mariorodriguez]

Hi @rajmehta53.

For this use-case, the best option is using the metrics-generator with span-metrics. This will create latency metrics of your spans, to which you can add attributes that are relevant for you, and then calculate the difference in latency.

[rajmehta53]

Is this available to play around with in the Grafana sandbox?
And would I need to instrument anything else apart from sending traces to Grafana?

[mariorodriguez]

Is this available to play around with in the Grafana sandbox?

No. These metrics are not being generated in that environment.

And would I need to instrument anything else apart from sending traces to Grafana?

Send them, and contact support so we can enable the feature for you.