# Otel collector with otlp exporter to Tempo issue

**URL:** <https://community.grafana.com/t/otel-collector-with-otlp-exporter-to-tempo-issue/120800>\
**Category:** Grafana Tempo\
**Created:** [May 2, 2024, 4:36pm UTC](https://community.grafana.com/t/otel-collector-with-otlp-exporter-to-tempo-issue/120800 "2024-05-02T16:36:33Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![danielnegreirosb](https://avatars.discourse-cdn.com/v4/letter/d/7ab992/32.png) [@danielnegreirosb](https://community.grafana.com/u/danielnegreirosb)\
**Post date:** [May 2, 2024, 4:36pm UTC](https://community.grafana.com/t/otel-collector-with-otlp-exporter-to-tempo-issue/120800/1 "2024-05-02T16:36:33Z")

</div>

Hi,

I have a Tempo on Kubernetes using helm chart [Get started with Grafana Tempo using the Helm chart | Grafana Labs Helm charts documentation](https://grafana.com/docs/helm-charts/tempo-distributed/next/get-started-helm-charts/)

I noticed the distributor is listening on 3100 and 9095  
level=info ts=2024-05-02T14:04:22.25847056Z caller=server.go:238 msg=“server listening on addresses” http=[::]:3100 grpc=[::]:9095

I configured the Tempo exporter like this for HTTP:

```
  otlphttp/tempo:
    endpoint: http://tempo-distributor:3100
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/apm, otlphttp/dynatrace, otlphttp/tempo]

```

But got the following error:  
Permanent error: rpc error: code = Unimplemented desc = error exporting items, request to [http://tempo-distributor:3100/v1/traces](http://tempo-distributor:3100/v1/traces) responded with HTTP Status Code 404", “dropped\_items”: 4}

And for GRPC i have the configuration like this

```
  otlp/tempo:
    endpoint: tempo-distributor:9095
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/apm, otlphttp/dynatrace, otlp/tempo]

```

but im getting the following error:

Permanent error: rpc error: code = Unimplemented desc = unknown service opentelemetry.proto.collector.trace.v1.TraceService", “dropped\_items”: 2}

I don’t think there is an error before or during the collector because it’s properly working for the other 2 trace exporters.

The PODs are all running  
tempo-compactor-77587cf9b5-lf5b2 1/1 Running 0 150m  
tempo-distributor-658b6499f7-ph84s 1/1 Running 0 150m  
tempo-ingester-0 1/1 Running 0 150m  
tempo-ingester-1 1/1 Running 0 150m  
tempo-ingester-2 1/1 Running 0 150m  
tempo-memcached-0 1/1 Running 0 150m  
tempo-querier-78885dd74f-76r2m 1/1 Running 0 150m  
tempo-query-frontend-7994df4bf-cmw24 1/1 Running 0 150m

Services:

tempo-compactor ClusterIP 10.233.54.245 3100/TCP 137m  
tempo-distributor ClusterIP 10.233.30.82 3100/TCP,9095/TCP 137m  
tempo-distributor-discovery ClusterIP None 3100/TCP 137m  
tempo-gossip-ring ClusterIP None 7946/TCP 137m  
tempo-ingester ClusterIP 10.233.11.79 3100/TCP,9095/TCP 137m  
tempo-ingester-discovery ClusterIP None 3100/TCP,9095/TCP 137m  
tempo-memcached ClusterIP 10.233.33.134 11211/TCP,9150/TCP 137m  
tempo-querier ClusterIP 10.233.28.188 3100/TCP,9095/TCP 137m  
tempo-query-frontend ClusterIP 10.233.18.32 3100/TCP,9095/TCP 137m  
tempo-query-frontend-discovery ClusterIP None 3100/TCP,9095/TCP,9096/TCP 137m

Does anyone know what I can do to fix this issue?

---

<div class="post-metadata">

**Author:** ![jangaraj](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/jangaraj/32/39197_2.png) [@jangaraj](https://community.grafana.com/u/jangaraj)\
**Post date:** [May 2, 2024, 6:17pm UTC](https://community.grafana.com/t/otel-collector-with-otlp-exporter-to-tempo-issue/120800/2 "2024-05-02T18:17:52Z")

</div>

Check if you have enabled otlp receiver:

> <https://github.com/monitoringartist/grafana-opentelemetry/blob/e8345860153ab7a6aad1227a7bec87487ec1de9f/tempo/tempo.yaml#L6>

If not, then find how to enable it in your helm values.

---

<div class="post-metadata">

**Author:** ![amangoworks](https://avatars.discourse-cdn.com/v4/letter/a/b3f665/32.png) [@amangoworks](https://community.grafana.com/u/amangoworks)\
**Post date:** [September 19, 2024, 11:29am UTC](https://community.grafana.com/t/otel-collector-with-otlp-exporter-to-tempo-issue/120800/3 "2024-09-19T11:29:30Z")

</div>

Can I send traces straight from opentelemetry.sdk to tempo-distributed-ingester port? or do I need a opentelemetry collector in between

---

<div class="post-metadata">

**Author:** ![jangaraj](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/jangaraj/32/39197_2.png) [@jangaraj](https://community.grafana.com/u/jangaraj)\
**Post date:** [September 19, 2024, 12:09pm UTC](https://community.grafana.com/t/otel-collector-with-otlp-exporter-to-tempo-issue/120800/4 "2024-09-19T12:09:21Z")

</div>

Yes, but that’s not a good practice. You may have increased latency and other problems on the app level - you will offload problems (batching, retrying, …) to OTEL collector usually.

---

<div class="post-metadata">

**Author:** ![amangoworks](https://avatars.discourse-cdn.com/v4/letter/a/b3f665/32.png) [@amangoworks](https://community.grafana.com/u/amangoworks)\
**Post date:** [September 19, 2024, 12:24pm UTC](https://community.grafana.com/t/otel-collector-with-otlp-exporter-to-tempo-issue/120800/5 "2024-09-19T12:24:37Z")

</div>

ok, but OTEL collector is still a push model instead of scraping from my service? So I think the increased problem for app still holds.  
It adds another hop, another point of failure

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/grafana/original/2X/e/ef5308cc58c9e5a13187b5012eb99f406928462f.png) [@system](https://community.grafana.com/u/system)\
**Post date:** [September 19, 2025, 12:24pm UTC](https://community.grafana.com/t/otel-collector-with-otlp-exporter-to-tempo-issue/120800/6 "2025-09-19T12:24:50Z")

</div>

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.
