OpenTelemetry Protocol (OTLP) Endpoint Credentials for Grafana Cloud

Grafana Cloud
1

I’m following this tutorial:
https://otlp-gateway-prod-us-central-0.grafana.net/otlp

First, I ran into 404s, because the endpoint information in the tutorial is out-of-date (discovered through this forum).

Now, I’m running into 401s. Given that the tutorial was already wrong about the endpoint, I’m wondering if it’s also wrong about the credentials. I’m using the instance ID of my Grafana Cloud instance, and a token configured as described by the tutorial, base64 encoding the two, and setting the Authorization header. Everything looks to me to be in order. Can somebody confirm whether instance ID:token is correct?

2

OTLP gateway in Grafana Cloud doesn’t have any SLA - there can be any problem.

Personally, I don’t use OTLP gateway, but Mimir, Tempo, Loki Grafana Cloud endpoints directly and I don’t have a problem.

3

This agent guide is more recent and shows how to use the Grafana Agent to receive via oltp and send to the Grafana backends.

This can also be done with the Open Telemetry collector.

4

The documentation pages should be up to date, but note that how things are configured might differ depending on what you are using. I’ll change the doc to make this clear.

When using the collector, you don’t need the final part of the URL, which is signal-specific. There, “https://otlp-gateway-prod-us-central-0.grafana.net/otlp” will work. In other places, when dealing with signal-specific configuration, you’ll need a suffix like “/v1/metrics” or “/v1/traces”.

From your message, it’s not clear to me what you are trying to configure, but I just tried the following Collector configuration against an account of mine and confirmed that it works:

extensions:
  basicauth:
    client_auth:
      username: "${env:GRAFANA_CLOUD_USER}"
      password: "${env:GRAFANA_CLOUD_TOKEN}"

processors:
  batch:

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  otlphttp:
    endpoint: https://otlp-gateway-prod-us-central-0.grafana.net/otlp
    auth:
      authenticator: basicauth

service:
  extensions: [ basicauth ]
  pipelines:
    traces:
      receivers:  [ otlp ]
      processors: [  ]
      exporters:  [ otlphttp ]

Note that GRAFANA_CLOUD_USER is the “Instance ID” that can be found under your “Grafana” settings for the stack, and GRAFANA_CLOUD_TOKEN is an API key with MetricsPublisher role.

I used telemetrygen (https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/telemetrygen) to send some traces to my collector:

telemetrygen traces --traces 100 --otlp-insecure

And the collector metrics confirmed that data was successfully sent:

$ curl -s localhost:8888/metrics | grep otelcol_exporter_sent_spans
# HELP otelcol_exporter_sent_spans Number of spans successfully sent to destination.
# TYPE otelcol_exporter_sent_spans counter
otelcol_exporter_sent_spans{exporter="otlphttp",service_instance_id="aa54b8a3-ddc5-49a2-a6e8-c03b916330a0",service_name="otelcol-contrib",service_version="0.87.0"} 200

And finally, I see traces on my Grafana Cloud account:

image
image1268×656 58.3 KB

2 Likes
5

I appreciate your detailed response, and apologize for misclassifying the documentation as out-of-date.

I didn’t mention what I was configuring, because I was afraid it would cloud the discussion, but now I’m wondering if it’s the problem. I am working with a serverless .Net application, and can’t really host the agent, so I’m trying to configure the OTLP Exporter to send directly to Grafana Cloud, something like this:

using var tracerProvider = Sdk.CreateTracerProviderBuilder()
	.AddOtlpExporter(o =>
	{
		o.Endpoint = new Uri("https://otlp-gateway-prod-us-central-0.grafana.net/otlp/v1/traces");
		o.Protocol = OtlpExportProtocol.HttpProtobuf;
		o.Headers = "Authorization=Basic " + auth64;
	})
6

How did you create auth64 variable? Are you sure, that’s correct (e. g. padding is correct)?

7

I can’t say that I’m certain:

var auth64 = Convert.ToBase64String(Encoding.ASCII.GetBytes($"{instanceId}:{token}"));

8

I would verify auth with postman first and use some online tools for base64 auth calculation

9

Without any changes, I’m now getting 200s back from the OTLP gateway (confirmed with Fiddler). I wonder if it just took a lot longer than the claimed 15 minutes for my token to become effective.

I still have a problem though, because I don’t see any traces in my Grafana instance.

10

It’s good that you are having 200s back from the OTLP gateway, but missing the traces is intriguing. Would you be able to enable the console exporter (or logging exporter, not sure how it’s called in .NET)? Having two exporters will help you see exactly what you are missing on Grafana Cloud, including the trace IDs.

11

It is called the console exporter in .NET, and I’ve gone ahead and enabled it. I see plenty of output that seems to correspond with the traces I see being sent to the OTLP gateway. My Grafana Cloud instance is new, and is currently empty of any data, so nothing is coming through.

12

It also shows that my token was just used under Access Policies:
image

13

Can you try the steps I used on my previous message, just to ensure there’s nothing wrong with your account? If you see traces from the telemetrygen tool, we can then focus on the SDK side instead.

14

I setup the collector, installed and ran telemetrygen, and ended up here:

> curl -s localhost:8888/metrics | grep otelcol_exporter_sent_spans
# HELP otelcol_exporter_sent_spans Number of spans successfully sent to destination.
# TYPE otelcol_exporter_sent_spans counter
otelcol_exporter_sent_spans{exporter="otlphttp",service_instance_id="d3141666-b7f7-4076-b3af-536c14195646",service_name="otelcol",service_version="0.87.0"} 400

I’m still seeing nothing in my Grafana instance.

The only difference in my setup is that otelcol couldn’t find the basicauth extension. I dug into this for a bit, but finally just set the headers property on the otlphttp exporter:

exporters:
  otlphttp:
    endpoint: https://otlp-gateway-prod-us-central-0.grafana.net/otlp
    headers: {"Authorization": "Basic mybase64encodedauthstring" }

I’m open to digging further into using the extension, but I’m thinking the metrics endpoint confirms that it sent something? Also, I ran it again with invalid credentials, and confirmed 0 traces going out.

15

That’s interesting, it does seem like data was sent. Do you have multiple stacks under your organization? If so, can you double-check you have the right one selected as the data source in the “Explore” tab? When going there, which service names do you see in the “Resource Service Name” field? For reference, here’s what I see:

image
image825×409 30 KB

1 Like