Drop spans with Grafana Agent

I’m fairly new to Tracing in general so it’s a big learning curve. We’re using Grafana Agent to receive traces being sent from Containers in our K8S cluster. Currently using OTLP.

But we’re seeing spans for what we want, but also for actuator/prometheus and actuator/health endpoints, that we don’t want to keep in Tempo.

Is it possible to configure Grafana Agent to drop spans that match a naming pattern like actuator.*?

Thanks!

Hello! Agent dev here. This will be possible in our Flow system we are developing in Agent. Flow will be coming out of experimental and into beta soon. The goal for tracing is to import most of the otel processors, I have created an issue Flow: `otelcol.processor.spanprocessor` component · Issue #2612 · grafana/agent · GitHub to track this if you would like to follow for when it is done.

Thanks @mattdurham

It’s such a basic thing to want to do, and given how easy it is to do drops for logs and metrics, I figured I was missing something obvious.

It may be possible now using a remote sampling document. This depends on whether or not your client supports the jaeger remote sampling doc.

All Jaeger clients and a few OTEL clients support this. I know Java and Go support this. Unsure on others, but it would allow you to set specific endpoints to have a 0% sampling rate.

@mattdurham I would say that filterprocessor will be better fit, because span needs to be dropped, not modified. Dropping of whole traces will be different story.