Error while sending Otel trace data to elastic endpoint via grafana-agent

  • What Grafana version and what operating system are you using?
    grafana-agent:v0.40.1_A0

  • What are you trying to achieve?
    Trying to use Grafana agent as a collector to receive otel trace data and export it to elasticsearch for storage.

  • How are you trying to achieve it?
    I have configured the grafana agent data pipeline in an openshift cluster to receive the otel data, (the ports for otel grpc/http are opened)

  • What happened?

When I send a sample trace I get the below error messages:

when using http in the url

	Error: Exporting failed. Will retry the request after interval." component=otelcol.exporter.otlphttp.elastic error="failed to make an HTTP request: Post \"http://elasticsearch:9200/v1/traces\": EOF" interval=10.73036973s

when using https in the url

	Error: Exporting failed. The error is not retryable. Dropping data." component=otelcol.exporter.otlphttp.elastic error="Permanent error: error exporting items, request to https://elasticsearch:9200/v1/traces responded with HTTP Status Code 406" dropped_items=1-removd colons around the url

the output on the cli for the sample trace:

curl -i https://grafana-agent-http-serverip.net/v1/logs -X POST -H "Content-Type: application/json" -d @span.json
HTTP/1.1 200 OK
content-type: application/json
date: Tue, 30 Apr 2024 18:32:27 GMT
content-length: 21
set-cookie: 6dbc74795e8e9b20ddcab6682a4dde93=0d5b23be19666d3e92c683ded09407f2; path=/; HttpOnly; Secure; SameSite=None

{"partialSuccess":{}}
  • What did you expect to happen?
    We expected the trace data to be exported to the elastic backend

  • Can you copy/paste the configuration(s) that you are having problems with?

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-agent-config
data:
  grafana-agent.river: |

    otelcol.exporter.otlphttp "elastic" {
      client {
        endpoint = "https://elasticsearch:9200"
        auth = otelcol.auth.basic.elastic.handler
        tls{
          ca_file = "/path/to/certs.pem"
        }
      }
    }

    otelcol.auth.basic "elastic" {
      username = "userid"
      password = "password"
    }

    otelcol.receiver.otlp "otel" {
      grpc {
        endpoint = "0.0.0.0:4317"
      }

      http {
        endpoint = "0.0.0.0:4318"
      }

      output {
        metrics = [otelcol.exporter.otlphttp.elastic.input]
        logs    = [otelcol.exporter.otlphttp.elastic.input]
        traces  = [otelcol.exporter.otlphttp.elastic.input]
      }
    }

    logging {
     level = "debug"
    }
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

  • Did you follow any online instructions? If so, what is the URL?

otelcol.receiver.otlp | Grafana Agent documentation

Collect OpenTelemetry data | Grafana Agent documentation

Are you sure that port 9200 is the right port for otlp data?

That’s the port our elastic team gave us to use as an endpoint. I was able to write a sample trace on 9200 from my local using curl. Should we open a specific port to receive the otlp data on elastic?

why not use this instead

Otel collector or Elastic Agents are not approved with our org security yet. My option is Grafana-agent or send the traces directly to elastic for now.

You sent some spans as logs (because you used /v1/logs endpoint, not /v1/traces) with your curl.

Ya, I tried both traces and logs just to see if they go as logs. But that fails too.

what is the syntax to add header to otelcol.exporter.otlphttp?