OpenTelemetry endpoint of Grafana Cloud

Grafana Cloud
1

Hello,

I was trying to configure my application to send data to Grafana Cloud using OpenTelemetry protocol according to instructions at Send data using OpenTelemetry Protocol (OTLP) | Grafana Cloud documentation. So I configured my application to publish to specified endpoint (https://otlp-gateway-prod-us-central-0.grafana.net/otlp) using specified protocol (http/protobuf) and specified authentication (Authorization=Basic base64(instace:key)). But for endpoint generated according to that page my application consistently gets 404 error.
Are there any hidden/missing settings which need to be turned on?

2

Hi @aleksandrkonstantino! I will attempt to help you troubleshoot the issue. Can you provide more details of your config and any logs you are seeing?

3

Hello,

Thanks a lot for response. I’m using OpenTelemetry Protocol Exporter with .NET code. The configuration adds exporter like this:
AddOtlpExporter(o =>
{
o.Endpoint = new Uri(cfg.Endpoint);
o.Protocol = OtlpExportProtocol.HttpProtobuf;
o.Headers = cfg.Headers;
});
}
Here cfg.Endpoint = “https://otlp-gateway-prod-us-central-0.grafana.net/otlp” and cfg.Headers = “Authorization=Basic NjE2…MwPQ==”.

When trying to report metrics and traces the code throws exception

System.Net.Http.HttpRequestException
HResult=0x80131500
Message=Response status code does not indicate success: 404 (Not Found).
Source=System.Net.Http
StackTrace:
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()

This exception was originally thrown at this call stack:
System.Net.Http.dll!System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() Unknown No symbols loaded.
OpenTelemetry.Exporter.OpenTelemetryProtocol.dll!OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ExportClient.BaseOtlpHttpExportClient<OpenTelemetry.Proto.Collector.Trace.V1.ExportTraceServiceRequest>.SendExportRequest(OpenTelemetry.Proto.Collector.Trace.V1.ExportTraceServiceRequest request, System.Threading.CancellationToken cancellationToken) Unknown No symbols loaded.
OpenTelemetry.Exporter.OpenTelemetryProtocol.dll!OpenTelemetry.Exporter.OtlpTraceExporter.Export(OpenTelemetry.Batch<System.Diagnostics.Activity> activityBatch) Unknown No symbols loaded.
OpenTelemetry.dll!OpenTelemetry.BatchExportProcessor<System.Diagnostics.Activity>.ExporterProc() Unknown No symbols loaded.
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown No symbols loaded.

4

Hello,
is there an update on the issue? I’m facing the same problem on the “prod-eu-west-0” instance.

5

From the stacktrace, this appears to be tracing instrumentation. Can you try appending /v1/traces to the URL?
https://otlp-gateway-prod-us-central-0.grafana.net/otlp/v1/traces

4 Likes
6

@robertlankford Thank you so much. This solved the issue for me.
After appending /v1/traces to the URL I got a “401 Unauthenticated” error because the library I use (opentelementry-otlp for Rust) does not yet support OTEL_EXPORTER_OTLP_HEADERS. Configuring everything in code solved that issue too, and now it works :slight_smile:

7

Works for me too. No more exceptions. And for metrics I added /v1/metrics.

8

I am also using Rust opentelemetry-otlp, but I am stuck with 401 Unauthenticated. Here is my config:

    let ot_auth = "Basic <base64(instance_id:key)>";
    let headers = HashMap::from([("Authorization".into(), ot_auth.into())]);
    let exporter = opentelemetry_otlp::new_exporter()
        .http()
        .with_endpoint("https://otlp-gateway-prod-us-east-0.grafana.net/otlp/v1/traces")
        .with_headers(headers);

Any idea what might be wrong @biwecka ?

9

Anybody managed to solve this? I am also stuck with a 401
@rdettai1

11

I ended up with setting the ‘OTEL_EXPORTER_OTLP_ENDPOINT’, ‘OTEL_EXPORTER_OTLP_HEADERS’, ‘OTEL_EXPORTER_OTLP_PROTOCOL’ environment variables like they are provided by Grafana in the cloud UI for the stack. Then i created a instance of the OTLPMetricExporter without any parameters. That did work and the solution is better than setting this in the code itself.

12

For Your Information, we have just refreshed the Grafana Cloud OTLP Endpoint documentation page to guide users in sending OTLP data to Grafana Cloud.

1 Like
13

Hello, I got this error "OpenTelemetry Javaagent failed to start
io.opentelemetry.sdk.autoconfigure.spi.ConfigurationException: Unsupported OTLP metrics protocol: “http/protobuf” when trying to run my java application with opentelemetry agent connect to OTLP endpoint. Does anyone here face this issue?
This my env config:
OTEL_SERVICE_NAME=“xxx”

OTEL_EXPORTER_OTLP_PROTOCOL=“http/protobuf”

OTEL_EXPORTER_OTLP_ENDPOINT=“xxx”

OTEL_EXPORTER_OTLP_HEADERS=“xxx”

14

Apparently your javaagent doesn’t support http/protobuf protocol.

15

I downloaded it from the gafana repo: Releases ¡ grafana/grafana-opentelemetry-java. So where can I find the support version?

16

The issue resolved. I change from “http/protobuf” → http/protobuf (remove double quoute from env variable)

1 Like