I’m using Otel SDKs to implement traces in an app. I’m also using pyroscope. How can I join the two together so that I get traces associated with given spans?
To connect traces and profiles using OpenTelemetry (Otel) SDKs and Pyroscope, you need to follow a series of steps to ensure that your application is properly instrumented and configured. Here's a step-by-step guide to help you achieve this:
- Instrument Your Application:
- Profiling: Ensure your application is instrumented with Pyroscope's SDK for the language you're using. For example, if you're using Python, you would use the Pyroscope Python SDK. Refer to the specific language guide for detailed instructions:
-
Tracing: Your application must also be instrumented with OpenTelemetry traces. Follow the OpenTelemetry guide for your language:
-
Configure the Client-Side Package:
-
Use the appropriate client-side package to link traces and profiles. This involves installing and configuring a package that connects the two telemetry signals (traces and profiles). For example:
- Python: Use
pyroscope-otel
package. - Go: Use
otel-profiling-go
package. - .NET: Use
Pyroscope.OpenTelemetry
package. - Java: Use
otel-profiling-java
package.
- Python: Use
-
Set Up the Tracer Provider:
-
For each language, you need to create and register a tracer provider that integrates with Pyroscope. This typically involves adding a span processor that links traces with profiling data.
-
Configure Grafana:
-
Ensure you have a Grafana instance running and configure the Tempo data source to discover linked traces and profiles. Follow the Tempo data source configuration documentation.
-
Link Traces to Profiles:
-
Use Grafana's "Trace to profiles" feature to link your Pyroscope data source to tracing data. This allows you to run queries from a trace span into the profile data using Grafana's Explore feature. You can configure this with either a basic configuration or a custom query.
-
Verify Configuration:
- Ensure that the
pyroscope.profile.id
key-value pair exists in your span tags. This is crucial for the "Profiles for this span" button and the embedded flame graph to be visible in Grafana.
By following these steps, you should be able to connect traces and profiles, allowing you to gain deeper insights into your application's performance and resource usage. If you encounter any issues, make sure to check the documentation for your specific language and setup.
grot is still learning and improving, please let me know how I did by reacting below.