How to create OTLP auth header OTEL_EXPORTER_OTLP_HEADERS for Grafana Cloud

Grafana Cloud
,
1

Hi,

I have followed the instructions to create an API Token and set the OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS environment variables, but I don’t get any data in grafana.net.

The same application successfully sends all telemetry to the Aspire dashboard.

How do I diagnose this?

2

Provide reproducible example, pls?

3

What do you mean? Give you access to my app and account?

4

If you are so brave, then yes - give everything to random guy on the internet :smiley:

No, reproducible example means how anyone can reproduce it. For example how did you instrument your app, what language are you using, which instructions did you followed, …

5

Of course I was not going to do that. :smile:

How I instrumented the application is irrelevant as I can see all telemetry on the Aspire dashboard.

But, if you want to try it, you can create a starter application and check it: Build your first .NET Aspire project - .NET Aspire | Microsoft Learn

6

That’s irrelevant, that’s simplistic, … I heard it many times :smiley: OK, so prove me that you have properly instrumented app. There is no point to investigate something at the end, when the source is not sending it.

This is Grafana forum. Don’t assume that someone have idea how Aspire dashboard works. Does it consume data from the same OTEL exporter as Grafana cloud? My blind guess is no, so this can be irrelevant :smiley:

1 Like
7

to say that to someone when they are trying to help you solve your problem is counter productive.

you can see it on Aspire but we cant

8

I followed the instructions here, to the best of my knowledge.

  • Set OTEL_EXPORTER_OTLP_PROTOCOL to http/protobuf
  • Set OTEL_EXPORTER_OTLP_ENDPOINT to https://otlp-gateway-prod-eu-west-2.grafana.net/otlp
  • Set OTEL_EXPORTER_OTLP_HEADERS to Authorization=Basic glc_…

With these settings I can see requests being sent to:

  • POST https://otlp-gateway-prod-eu-west-2.grafana.net/otlp/v1/logs HTTP/1.1
  • POST https://otlp-gateway-prod-eu-west-2.grafana.net/otlp/v1/traces HTTP/1.1
  • POST https://otlp-gateway-prod-eu-west-2.grafana.net/otlp/v1/metrics HTTP/1.1

But they all get a 401 reply:

Request:

POST https://otlp-gateway-prod-eu-west-2.grafana.net/otlp/v1/logs HTTP/1.1
Host: otlp-gateway-prod-eu-west-2.grafana.net
Authorization: Basic glc_...
User-Agent: OTel-OTLP-Exporter-Dotnet/1.9.0
Content-Type: application/x-protobuf
Content-Length: 157968

...

Response:

HTTP/1.1 401 Unauthorized
Content-Type: application/json
X-Content-Type-Options: nosniff
Date: Fri, 09 Aug 2024 18:33:00 GMT
Content-Length: 74

{"status":"error","error":"authentication error: no credentials provided"}

Tried 3 tokens, all with the scopes:

  • alerts:write
  • logs:write
  • metrics:write
  • rules:write
  • traces:write
  • profiles:write
  • metrics:import
  • datadog:validate

All 3 tokens report in the access policies portal that have never been used.

2 Likes
9

Lovely problem description.

I guess you have wrong auth Authorization=Basic glc_
It must be Authorization=Basic <credentials>, where credentials are base64(username:accesstoken). It looks like you are sending as credentials only access token.

10

glc_ is the beginning of the credentials given in the instructions where the token was created.

It’s, actually, JSON base64-encoded.

11

:person_shrugging: :person_shrugging: :person_shrugging: :person_shrugging:

Now it’s time to share your credentials, pls. What was used in that webinar?

image
image1421×1136 119 KB

Rewrite that NDY... value into text and make base64 decode. What will be the result?

12

The parte after glc_ is Base64-encoded JSON with the following content:

{"o":"ooooooo","n":"stack-sssssss-otlp-write-stack-sssssss-otlp-write","k":"kkkkkkk","m":{"r":"prod-eu-west-2"}}

Where:

  • ooooooo
  • sssssss is the stack ID
  • stack-sssssss-otlp-write-stack-sssssss-otlp-write is the name of the token
  • kkkkkkk is, probably, the access key
13

Sorry, you don’t understand. Follow pictures:

obrázok
obrázok2229×1723 304 KB

API token (BTW it’s JWT read RFC 7519):

glc_eyJvIjoiMTI5MzgiLCJuIjoic3RhY2stMzk4MTItb3RscC13cml0ZS1kZWxtZSIsImsiOiJncTRNZlREdU02WmFMMElmMDk4VDkyMDQiLCJtIjp7InIiOiJ1cyJ9fQ==

Instance ID:

39812

So basic auth header will be base64(instance ID:API token) - read RFC 7617:

obrázok
obrázok2306×1811 354 KB

So what will be basic auth header?

obrázok
obrázok2046×880 113 KB

What was used in the referred webinar?

obrázok
obrázok1920×1526 172 KB

So what will you get when you use API token instead of base64(instance ID:API token) in basic header?

HTTP/1.1 401 Unauthorized
Content-Type: application/json
X-Content-Type-Options: nosniff
Date: Fri, 09 Aug 2024 18:33:00 GMT
Content-Length: 74

{"status":"error","error":"authentication error: no credentials provided"}

Which is correct, because you are not sending correct auth.

Does it makes sense?

14

OK!

Now I got what the issue was. I though the token was all that was needed.

That website might be pretty, but it’s very hard to read for some people. And if you do not create the token there, there’s no hint of what you explained to be found.

By the way, a JWT is a bit more that Base64 URL encoded (not just Base64 encoded) JSON.

Thanks for your help.

15

OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic <base64 instanceID:token>

I would say doc is your friend.