Error while sending Otel trace data to elastic endpoint via grafana-agent

Grafana Alloy
1

  • What Grafana version and what operating system are you using?
    grafana-agent:v0.40.1_A0

  • What are you trying to achieve?
    Trying to use Grafana agent as a collector to receive otel trace data and export it to elasticsearch for storage.

  • How are you trying to achieve it?
    I have configured the grafana agent data pipeline in an openshift cluster to receive the otel data, (the ports for otel grpc/http are opened)

  • What happened?

When I send a sample trace I get the below error messages:

when using http in the url

	Error: Exporting failed. Will retry the request after interval." component=otelcol.exporter.otlphttp.elastic error="failed to make an HTTP request: Post \"http://elasticsearch:9200/v1/traces\": EOF" interval=10.73036973s

when using https in the url

	Error: Exporting failed. The error is not retryable. Dropping data." component=otelcol.exporter.otlphttp.elastic error="Permanent error: error exporting items, request to https://elasticsearch:9200/v1/traces responded with HTTP Status Code 406" dropped_items=1-removd colons around the url

the output on the cli for the sample trace:

curl -i https://grafana-agent-http-serverip.net/v1/logs -X POST -H "Content-Type: application/json" -d @span.json
HTTP/1.1 200 OK
content-type: application/json
date: Tue, 30 Apr 2024 18:32:27 GMT
content-length: 21
set-cookie: 6dbc74795e8e9b20ddcab6682a4dde93=0d5b23be19666d3e92c683ded09407f2; path=/; HttpOnly; Secure; SameSite=None

{"partialSuccess":{}}

  • What did you expect to happen?
    We expected the trace data to be exported to the elastic backend

  • Can you copy/paste the configuration(s) that you are having problems with?

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-agent-config
data:
  grafana-agent.river: |

    otelcol.exporter.otlphttp "elastic" {
      client {
        endpoint = "https://elasticsearch:9200"
        auth = otelcol.auth.basic.elastic.handler
        tls{
          ca_file = "/path/to/certs.pem"
        }
      }
    }

    otelcol.auth.basic "elastic" {
      username = "userid"
      password = "password"
    }

    otelcol.receiver.otlp "otel" {
      grpc {
        endpoint = "0.0.0.0:4317"
      }

      http {
        endpoint = "0.0.0.0:4318"
      }

      output {
        metrics = [otelcol.exporter.otlphttp.elastic.input]
        logs    = [otelcol.exporter.otlphttp.elastic.input]
        traces  = [otelcol.exporter.otlphttp.elastic.input]
      }
    }

    logging {
     level = "debug"
    }

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

  • Did you follow any online instructions? If so, what is the URL?

otelcol.receiver.otlp | Grafana Agent documentation

Collect OpenTelemetry data | Grafana Agent documentation

2

Are you sure that port 9200 is the right port for otlp data?

3

That’s the port our elastic team gave us to use as an endpoint. I was able to write a sample trace on 9200 from my local using curl. Should we open a specific port to receive the otlp data on elastic?

4

image
image1742×1070 146 KB

why not use this instead

5

Otel collector or Elastic Agents are not approved with our org security yet. My option is Grafana-agent or send the traces directly to elastic for now.

6

You sent some spans as logs (because you used /v1/logs endpoint, not /v1/traces) with your curl.

7

Ya, I tried both traces and logs just to see if they go as logs. But that fails too.

8

what is the syntax to add header to otelcol.exporter.otlphttp?