Thanks for this pointer. I will be taking a look into it shortly.
I also tried using the Grafana agent. I am able to send the traces there (using gRPC), and it’s able to write them to my local tempo instance, but I get a 401 when trying to write to my Grafana Cloud instance:
2023-10-03 09:51:31 ts=2023-10-03T07:51:31.73460844Z caller=zapadapter.go:84 level=error component=traces traces_config=default kind=exporter data_type=traces name=otlp/1 msg="Exporting failed. The error is not retryable. Dropping data." error="Permanent error: rpc error: code = Unauthenticated desc = unexpected HTTP status code received from server: 401 (Unauthorized); transport: received unexpected content-type \"application/json\"" dropped_items=10 2023-10-03 09:51:32 ts=2023-10-03T07:51:32.630559006Z caller=zapadapter.go:84 level=error component=traces traces_config=default kind=exporter data_type=traces name=otlp/1 msg="Exporting failed. The error is not retryable. Dropping data." error="Permanent error: rpc error: code = Unauthenticated desc = unexpected HTTP status code received from server: 401 (Unauthorized); transport: received unexpected content-type \"application/json\"" dropped_items=1
This I get with the following configuration, which is described on the docs like so:
server:
log_level: info
traces:
configs:
- name: default
receivers:
otlp:
protocols:
http:
grpc:
remote_write:
- endpoint: tempo:4317
insecure: true
- endpoint: tempo-prod-08-prod-eu-west-3.grafana.net:443
basic_auth:
username: myusernamehere
password: mypasswordhere
batch:
timeout: 5s
send_batch_size: 100
It is a fresh token copy / pasted, so it should be fine. Any idea why this might be?