Configuring Otel Java SDK to send traces direct to grafana-cloud

Grafana Cloud
1

I had a lot of trouble understanding how to get this going, so I thought I would post this here.

Go into the account-ui (not the stack-ui) and click “Open Telemetry”.

This shows you your instance-id and tells you the oltp endpoint
https://otlp-gateway-prod-us-east-0.grafana.net/otlp, which is wrong for
sending traces, but whatever.

On this page, you must also create they api-key (save it, no way to see it
later, I think?)

To construct the correct url for Otel SDK to send traces, you need to add
/v1/traces to the end of the oltp endpoint url:
https://otlp-gateway-prod-us-east-0.grafana.net/otlp/v1/traces.
Use /otlp/v1/metrics for directly sending metrics.

I only figured this out thanks to this topic:

From there, configure a new OtlpHttpSpanExporter and create a standard
Authorization header from the instance-id:api-key pair.

At this point, OTel started sending traces that I could see in the stack-ui
by navigating to / Explore / Traces.

      String authPair = otelProps.grafanaCloudUser() + ":" +
        otelProps.grafanaCloudPassword();
      String basicAuthHeader = "Basic " +
        Base64.getEncoder().encodeToString(authPair.getBytes());

      var spanExporter = OtlpHttpSpanExporter.builder().
        setEndpoint(grafana.url()).
        addHeader("Authorization", basicAuthHeader).
        build();

      config.addSpanProcessor(BatchSpanProcessor.builder(spanExporter).
        setMaxQueueSize(grafana.maxQueueSize()).
        setMaxExportBatchSize(grafana.maxBatchSize()).
        setScheduleDelay(grafana.scheduleDelay()).
        setExporterTimeout(grafana.exportTimeout()).
        build()
      );
2

See OpenTelemetry Protocol Exporter | OpenTelemetry

https://otlp-gateway-prod-us-east-0.grafana.net/otlp/ is

Endpoint (OTLP/HTTP): Target URL to which the exporter is going to send spans, metrics, or logs.

So https://otlp-gateway-prod-us-east-0.grafana.net/otlp/ is OTEL_EXPORTER_OTLP_ENDPOINT and

When using OTEL_EXPORTER_OTLP_ENDPOINT, exporters MUST construct per-signal URLs as described below.
…
If signals are sent that have no per-signal configuration from the previous point, OTEL_EXPORTER_OTLP_ENDPOINT is used as a base URL and the signals are sent to these paths relative to that:

  • Traces: v1/traces
  • Metrics: v1/metrics.
  • Logs: v1/logs.

You are using the low-level code, so you have to construct the correct URL in your case. If you are using something ready, e.g., opentelemetry java agent autoinstrumentation, that will automatically append correct relative path to base URL.

Grafana doc is correct - it doesn’t claim that https://otlp-gateway-prod-us-east-0.grafana.net/otlp/ is for traces specific. That’s only base OTEL URL.