Error when exporting OTLP data from Grafana Alloy to Grafana Cloud

Grafana Cloud
, ,
1

Hi everyone,

I have been hitting my head on the keyboard for the last few days trying to send trace signals from my TypeScript HTTP server instrumented with the OpenTelemetry SDK to the Grafana Cloud using Grafana Alloy as a collector, but it doesn’t work.

The server is generating the traces as expected and I was able to see them on Jaeger and Tempo running locally as containers.

I installed alloy and updated the configuration for the following using the info provided here:

// Sample config for Alloy.
//
// For a full configuration reference, see https://grafana.com/docs/alloy
logging {
  level = "warn"
}

otelcol.receiver.otlp "otlp_receiver" {
  grpc {
    endpoint = "127.0.0.1:4317"
  }
  http {
    endpoint = "127.0.0.1:4318"
  }

  output {
    traces = [otelcol.exporter.otlp.grafanacloud.input,]
  }
}

otelcol.exporter.otlp "grafanacloud" {
  client {
    endpoint = "tempo-prod-17-prod-sa-east-1.grafana.net:443"
    auth = otelcol.auth.basic.grafanacloud.handler
  }
}

otelcol.auth.basic "grafanacloud" {
  username = "11..."
  password = "glc_eyJ..."
}

And my instrumentation file in TypeScript is the following:

import { AwsInstrumentation } from '@opentelemetry/instrumentation-aws-sdk';
import { HttpInstrumentation } from '@opentelemetry/instrumentation-http';
import { MongoDBInstrumentation } from '@opentelemetry/instrumentation-mongodb';
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { Resource } from '@opentelemetry/resources';
import { RuntimeNodeInstrumentation } from '@opentelemetry/instrumentation-runtime-node';

import info from '../package.json';

const sdk = new NodeSDK({
  instrumentations: [
    new AwsInstrumentation(),
    new HttpInstrumentation(),
    new MongoDBInstrumentation({ dbStatementSerializer: () => '' }),
    new RuntimeNodeInstrumentation({ monitoringPrecision: 1_000 }),
  ],
  resource: new Resource({
    'service.name': info.name,
    'service.version': info.version,
  }),
  traceExporter: new OTLPTraceExporter({}),
});

sdk.start();

When I run the server and check the Alloy log, I have the following error message: evel=error msg="Exporting failed. Dropping data." component_path=/ component_id=otelcol.exporter.otlp.grafanacloud error="not retryable error: Permanent error: rpc error: code = Unauthenticated desc = unexpected HTTP status code received from server: 401 (Unauthorized); transport: received unexpected content-type \"application/json\"" dropped_items=3.

1 Like
2

This indicates wrong credentials. Double check them + if you have correct permissions (for example you can have correct credentials, but only for metrics reading - they will fail, when you will try to use them for metric write).

3

This indicates a problem with used encoding/protocol. Grafana cloud supports http/protobuf, so otelcol.exporter.otlphttp should be used, no otelcol.exporter.otlp.