OTLP from application to Grafana Cloud not workng

Grafana Authentication
1

The code used and logs are found here

  • What Grafana version and what operating system are you using?

    • Grafana Cloud, sending traces using OTLP with Rust.

    Using a local tempo docker image with the above code works, the only thing I change is set this env var like so: OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"

  • What are you trying to achieve?
    Send data using OTLP from my application to Grafana cloud.

  • How are you trying to achieve it?
    Following the docs here:
    Send data using OpenTelemetry Protocol (OTLP) | Grafana Cloud documentation
    I also tried using the instance ID of both grafana and tempo in my account.

  • What happened?
    See logs in gist. connection error detected: frame with invalid size (line=165,target=hyper::proto::h2::client)

  • What did you expect to happen?
    Traces to send successfully and start showing up in Grafana Cloud Tempo.

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    No.

The code used and logs are found at the top of this post.

2

Error message indicates that exporter tried to make grpc connection. Grafana cloud supports only http,not grpc, so grpc connection can’t be successful.

Are you sure that used OTEL Rust lib has proper support for OTEL_EXPORTER_OTLP_PROTOCOL env variable?

I would say that you have to use https://crates.io/crates/opentelemetry-http for http.

3

Thanks for this pointer. I will be taking a look into it shortly.

I also tried using the Grafana agent. I am able to send the traces there (using gRPC), and it’s able to write them to my local tempo instance, but I get a 401 when trying to write to my Grafana Cloud instance:

2023-10-03 09:51:31 ts=2023-10-03T07:51:31.73460844Z caller=zapadapter.go:84 level=error component=traces traces_config=default kind=exporter data_type=traces name=otlp/1 msg="Exporting failed. The error is not retryable. Dropping data." error="Permanent error: rpc error: code = Unauthenticated desc = unexpected HTTP status code received from server: 401 (Unauthorized); transport: received unexpected content-type \"application/json\"" dropped_items=10 2023-10-03 09:51:32 ts=2023-10-03T07:51:32.630559006Z caller=zapadapter.go:84 level=error component=traces traces_config=default kind=exporter data_type=traces name=otlp/1 msg="Exporting failed. The error is not retryable. Dropping data." error="Permanent error: rpc error: code = Unauthenticated desc = unexpected HTTP status code received from server: 401 (Unauthorized); transport: received unexpected content-type \"application/json\"" dropped_items=1

This I get with the following configuration, which is described on the docs like so:

image
image1051×392 23.4 KB

server:
  log_level: info

traces:
  configs:
    - name: default
      receivers:
        otlp:
          protocols:
            http:
            grpc:
      remote_write:
        - endpoint: tempo:4317
          insecure: true
        - endpoint: tempo-prod-08-prod-eu-west-3.grafana.net:443
          basic_auth:
            username: myusernamehere
            password: mypasswordhere
      batch:
        timeout: 5s
        send_batch_size: 100

It is a fresh token copy / pasted, so it should be fine. Any idea why this might be?

4

Thanks again for this steer. I updated the code to use the HTTP exporter accordingly. I added a new file with the code in the gist.

As instructed by the docs, the endpoint was set to:
OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp-gateway-prod-eu-west-3.grafana.net/otlp"

And I get the following error:

file: /home/liam/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.27/src/client/connect/dns.rs [2023-10-03T10:00:53.207Z] DEBUG: veloxide-server-localdev/32073 on PadCore: connecting to 20.31.17.143:443 (line=537,target=hyper::client::connect::http) file: /home/liam/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.27/src/client/connect/http.rs [2023-10-03T10:00:53.221Z] DEBUG: veloxide-server-localdev/32073 on PadCore: connected to 20.31.17.143:443 (line=540,target=hyper::client::connect::http) file: /home/liam/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.27/src/client/connect/http.rs [2023-10-03T10:00:53.245Z] DEBUG: veloxide-server-localdev/32073 on PadCore: flushed 3375 bytes (line=342,target=hyper::proto::h1::io) file: /home/liam/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.27/src/proto/h1/io.rs [2023-10-03T10:00:53.267Z] DEBUG: veloxide-server-localdev/32073 on PadCore: parsed 6 headers (line=207,target=hyper::proto::h1::io) file: /home/liam/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.27/src/proto/h1/io.rs [2023-10-03T10:00:53.267Z] DEBUG: veloxide-server-localdev/32073 on PadCore: incoming body is content-length (19 bytes) (line=222,target=hyper::proto::h1::conn) file: /home/liam/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.27/src/proto/h1/conn.rs [2023-10-03T10:00:53.268Z] DEBUG: veloxide-server-localdev/32073 on PadCore: incoming body completed (line=298,target=hyper::proto::h1::conn) file: /home/liam/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.27/src/proto/h1/conn.rs [2023-10-03T10:00:53.268Z] DEBUG: veloxide-server-localdev/32073 on PadCore: pooling idle connection for ("https", otlp-gateway-prod-eu-west-3.grafana.net) (line=376 ,target=hyper::client::pool) file: /home/liam/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.27/src/client/pool.rs OpenTelemetry trace error occurred. request failed with status 404 Not Found

I also tried removing the /otlp from the endpoint at the end, but get the same result.

1 Like
5

Try /otlp/ path, pls.

BTW I wouldn’t use otlp gateway in prod, because it doesn’t have SLA.
I would export traces directly to Tempo. It will be more reliable from the SLA perspective. It has support for GRPC, so it will be also faster solution.

6

Trying with /otlp/ I get the same issue.

I’d like to use the grafana agent (whatever works honestly), but that isn’t working either, as described in my comment above.

7

Try
https://otlp-gateway-prod-eu-west-3.grafana.net/otlp/v1/traces