How to pass x-scope-orgid dynamically in otel collector while using tempo in multitenant mode?

[apoorvgupta772]

I am trying to send traces to tempo in multitenant mode by using Zipkin exporter and otel collector.
I am able to do that successfully but the problem is each time I am not able to pass x-scope-orgId dynamically . To make it more clear I am enclosing my configs.

This is my otel-collector-config.yaml file →

receivers:
zipkin:

processors:
batch:

exporters:
otlp:
endpoint: tempo:4317
tls:
insecure: true
headers:
x-scope-orgid: 123 || Here I want to pass orgId dynamically
service:
pipelines:
traces:
receivers: [zipkin]
processors: [batch]
exporters: [otlp]

How to pass orgId dynamically ?

[joeelliott]

I’m not sure there’s a great answer here. One that comes to mind is the routing processor:

opentelemetry-collector-contrib/processor/routingprocessor at main ·...

If you can reliably use an attribute on a span to “choose” the tenant then you could route to an exporter that is configured appropriately.

[mariorodriguez]

I believe the headers_setter extension is what you need.

[apoorvgupta772]

@mariorodriguez @joeelliott Thanks for answering
@mariorodriguez I have tried with this but whenever I am using this extension the docker file for otel collector getting stopped.

This is otel-collector-config.yml that i am using now

extensions:
headers_setter:
headers:
- key: X-Scope-OrgId
from_context: tenant_id

receivers:
zipkin:

processors:
nop:

exporters:
otlp:
endpoint: tempo:4317
insecure: true
auth:
authenticator: headers_setter
service:
extensions: [ headers_setter ]
pipelines:
traces:
receivers: [zipkin]
processors: [nop]
exporters: [otlp]

[joeelliott]

Is it logging any errors?

Also, make sure you’re using this image Docker Hub

since those are contrib processors

[apoorvgupta772]

@joeelliott It is just exiting . I am enclosing image for better understanding. tried with the docker image you have suggested.
after adding headers_setter , the docker image just gets exited.

[apoorvgupta772]

@mariorodriguez @joeelliott
1- Is there any other way to pass X-Scope-OrgId dynamically ?
2- How can we solve that otel collector gets exited while using header_setter extension.
3- Can someone directly send traces to tempo using a grpc request without using otel collector to use in a multitenant setup?

[mariorodriguez]

1- Is there any other way to pass X-Scope-OrgId dynamically ?

Yes, using the headers_setter extension

2- How can we solve that otel collector gets exited while using header_setter extension.

This will get fixed with v0.59.0 of the OTel collector, which should be available now.

3- Can someone directly send traces to tempo using a grpc request without using otel collector to use in a multitenant setup?

Yes, but making gRPC requests manually is very tedious. You can use a tool like grpcurl.

[apoorvgupta772]

@mariorodriguez Thanks for answering but even after using the v0.59.0 , the otelcollector getting exited after adding headers_extension … if using extension like zpages and health_check then otel collector is not exiting. Do I have to change something in configs?

[apoorvgupta772]

@joeelliott @mariorodriguez
this is otel-collector-config.yaml

extensions:
headers_setter:
headers:
- key: X-Scope-OrgId
from_context: X-Scope-OrgId
receivers:
zipkin:

exporters:
otlp:
endpoint: tempo:4317
tls:
insecure: true
# insecure_skip_verify: true
auth:
authenticator: headers_setter

service:
extensions: [ headers_setter ]
pipelines:
traces:
receivers: [zipkin]
exporters: [otlp]

for this config docker file of otel-collector gets exit while adding headers_setter as extension and It is logging this error

if removing insecure:true or auth: authenticator:headers_setter, it is not exiting initially.

[mariorodriguez]

That happens because the headers_setter extension can’t work with an insecure connection. Could you enable TLS?

[apoorvgupta772]

@mariorodriguez Yes I have already added
tls:
insecure:true in exporter like this still it is showing this error.

exporters:
otlp:
endpoint: tempo:4317
tls:
insecure: true
auth:
authenticator: headers_setter

[mariorodriguez]

The headers_setter extension does not work over insecure connections.

tls:
  insecure: true

That config is defining that exporting will be done over an insecure connection (without TLS), thus it won’t work.

[apoorvgupta772]

@mariorodriguez okay thanks , it means I have to create tls certificates?

[mariorodriguez]

It appears so. I’d try to reach out to the OTel collector community to verify the TLS requirements. It seems suboptimal.

[apoorvgupta772]

@mariorodriguez @joeelliott I have created tls certificates and they are perfectly fine.
In recievers section of otel collector when I am using lnclude metadata true like this.

receivers:
zipkin:
protocols:
http:
include_metadata: true
then this error is coming

Below is my otel-collector-config.yaml is

extensions:
headers_setter:
headers:
- key: X-Scope-OrgId
from_context: X-Scope-OrgId
receivers:
zipkin:
protocols:
http:
include_metadata: true

exporters:
otlp:
endpoint: tempo:4317
tls:
cert_file: /etc/cert.pem
key_file: /etc/cert-key.pem
insecure_skip_verify: true
auth:
authenticator: headers_setter

service:
extensions: [ headers_setter ]
pipelines:
traces:
receivers: [zipkin]
exporters: [otlp]

looks like it is not the right way to pass metadata. can you please tell about this error?

P.S. I am using otel/opentelemetry-collector-contrib:0.59.0 for otel collector and grafana/tempo:latest for tempo.

[mariorodriguez]

Check the zipking receiver configuration, it doesn’t have a protocols block.

[apoorvgupta772]

Firstly, Thanks @mariorodriguez @joeelliott for helping as Now I am able to push traces.

Basically when I am generating traces using otlp Trace Exporter and using otlp as a reciever and passing in config like this
receivers:
otlp:
protocols:
http:
include_metadata: true

headers_setters extension is perfectly working fine and data is getting pushed in tempo in multitenant mode . but when I am using zipkin exporter for creating traces and zipkin as a reciever then traces are not pushing in tempo in multitenant mode through headers_setter_extension . I guess metadata is not passing.
For zipkin,
receivers:
zipkin:
I am using like this.

What change should I do so that it will work fine with zipkin too?