Is it possible to turn off tracing?

[ragamx]

Hi All

In our system we are using opencensus to instrument our code and Tempo as backend. Due to memory restrictions in some cases we don’t want to have the backend running. However if it is not running the code will log errors messages, is there a way to avoid these messages by turning off tracing or something else?

Thanks

[koenraad]

This will depend on the library you are using to trace your application. A lot of libraries have a setting to disable sending traces or you can set sampling at 0.

What are you using to send traces to Tempo?

[ragamx]

We are using opencensus with zipkin configuration.

[koenraad]

I haven’t used opencensus myself before, so I’d recommend asking this question again in their forums or chat channels. They might know a configuration to help you out.

If you don’t mind running an additional component in between your application and Tempo, you can setup a tracing pipeline with the OpenTelemetry Collector.
It’s possible to run the probabilistic sampler processor between the receiver and the exporter. If you set the sampling percentage at 0 this will drop all traces.
This would allow you to control this in a central place.

Your config would look something like this:

receivers:
  zipkin:
processors:
  probabilistic_sampler:
    hash_seed: 22
    sampling_percentage: 100  # set this to 0 to drop all traces
exporters:
  zipkin:
    endpoint: tempo:9411     # replace with your endpoint, you could also export traces using OTLP GRPC
    insecure: true           # TODO check if necessary
service:
  pipelines:
    traces:                          # This pipeline will...
      receivers: [zipkin].              # ...receive zipkin traces
      processors: [probabilistic_sampler]  # ...sample them
      exporters: [zipkin]              # and export zipkin traces to Tempo

Refer to the OpenTelemetry documentation for more details: Configuration | OpenTelemetry

[koenraad]

A colleague (@mariorodriguez) shared this solution using the Grafana Agent. The Grafana Agent wraps the OpenTelemetry Collector but is more opinionated about using the Grafana stack. We also offer some nice extra features like automatic logging and generating metrics from trace data.

The config will look something like this:

tempo:
  configs:
  - name: default
    receivers:
      zipkin:
    remote_write:  # this uses OTLP GRPC which will have to be enabled in Tempo
      - endpoint: tempo:4317
    tail_sampling:
      decision_wait: 1s
      policies:
        - rate_limiting:
            spans_per_second: 0  # at 0 spans / second, all traces will be dropped as well

See the config docs: tempo_config | Grafana Labs

[ragamx]

I have tried to get some info from one of the opencensus community but nothing yet. I will try the sampling percentage option.

Thank you @koenraad