Send OpenTelemetry metrics to DataDog

Grafana k6
1

Hi Team,
As per the latest version of k6, i updated the version and trying to send a k6 metrics to datadog by using the following command and document. As per this document i didn’t see any data send to datadog

$  K6_OTEL_GRPC_EXPORTER_INSECURE=true K6_OTEL_METRIC_PREFIX=k6_ k6 run --tag test-id=123 -o experimental-opentelemetry examples/script.js```

https://grafana.com/docs/k6/latest/results-output/real-time/opentelemetry/

I saw this documentaion help to resolve that wrong percentiles values issues
https://github.com/grafana/k6/issues/2044#issuecomment-2194455469

how do i connect to otlp and send k6 metrics to datadog
Can any one help here?
2

Hey @Naveen,
You are not providing the DataDog Agent’s endpoint where you want to send the metrics.

You have to set the K6_OTEL_GRPC_EXPORTER_ENDPOINT or K6_OTEL_HTTP_EXPORTER_ENDPOINT option for it. If you don’t do that then the default value is used as documented on the OpenTelemetry | Grafana k6 documentation page.

I hope it helps

3

Do you mean, I need to set that Datadog agent IP address where it is running?

4

Yes, otherwise the metric output doesn’t know where to flush the data.

It might be worth reading this DataDog documentation’s pages OpenTelemetry in Datadog and OTLP Ingestion by the Datadog Agent.

5

Getting the following error while executing this command

K6_OTEL_HTTP_EXPORTER_ENDPOINT=http://xxx.xx.x.x:4318 K6_OTEL_METRIC_PREFIX=k6_ k6 run --tag test-id=Perf123 -o experimental-opentelemetry scripts/test.js

/\ |‾‾| /‾‾/ /‾‾/

/\ / \ | |/ / / /

/ / \ | ( / ‾‾\

/ \ | |\ \ | (‾) |

/ __________ \ |__| _\ ____/ .io

execution: local

script: scripts/test.js

output: opentelemetry (grpc, localhost:4317)

scenarios: (100.00%) 1 scenario, 5 max VUs, 4m30s max duration (incl. graceful stop):

  • default: Up to 5 looping VUs for 4m0s over 3 stages (gracefulRampDown: 30s, gracefulStop: 30s)

2024/08/26 17:06:23 failed to upload metrics: context deadline exceeded: rpc error: code = Unavailable desc = connection error: desc = “transport: authentication handshake failed: tls: first record does not look like a TLS handshake”

2024/08/26 17:06:33 failed to upload metrics: context deadline exceeded: rpc error: code = Unavailable desc = connection error: desc = “transport: authentication handshake failed: tls: first record does not look like a TLS handshake”

6

As per your inputs, while executing the below command I’m getting the below error

command:

K6_OTEL_GRPC_EXPORTER_INSECURE=true K6_OTEL_GRPC_EXPORTER_ENDPOINT=http://111.11.0.1:4317 K6_OTEL_METRIC_PREFIX=k6_ k6 run --tag test-id=Perf123 -o experimental-opentelemetry scripts/test.js

Error:

ERRO[0011] can’t shutdown OpenTelemetry metric provider error=“failed to upload metrics: context deadline exceeded: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: address http://111.11.0.1:4317: too many colons in address"”

Can you please help me here?

7

If you are using HTTP then you have to use the K6_OTEL_HTTP_EXPORTER_ENDPOINT and not the gRPC. Furthermore, you’re adding the not required http:// scheme of the protocol.
The correct format should be:

K6_OTEL_HTTP_EXPORTER_ENDPOINT=111.11.0.1:4317

Let me know if it helps

8

As per your suggestion, I have updated the command and executed it but getting the following error. can you please help me here?

command:

K6_OTEL_HTTP_EXPORTER_INSECURE=true K6_OTEL_EXPORT_INTERVAL=120s K6_OTEL_HTTP_EXPORTER_ENDPOINT=111.11.0.1:4317 K6_OTEL_METRIC_PREFIX=k6_ k6 run --tag test-id=Perf123 -o experimental-opentelemetry scripts/test.js

Error:

ERRO[0011] can’t shutdown OpenTelemetry metric provider error=“failed to upload metrics: context deadline exceeded: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"”

9

Hi @codebien, As per your suggestion, I have updated the command and executed it but getting the following error. can you please help me here?

command:

K6_OTEL_HTTP_EXPORTER_INSECURE=true K6_OTEL_EXPORT_INTERVAL=120s K6_OTEL_HTTP_EXPORTER_ENDPOINT= 111.11.0.1:4317 K6_OTEL_METRIC_PREFIX=k6_ k6 run --tag test-id=Perf123 -o experimental-opentelemetry scripts/test.js

Error:

ERRO[0011] can’t shutdown OpenTelemetry metric provider error=“failed to upload metrics: context deadline exceeded: rpc error: code = Unavailable desc = connection error: desc = “transport: authentication handshake failed: tls: first record does not look like a TLS handshake””

10

It sounds like you have misconfigured the DataDog agent. Make sure the agent doesn’t expect to get a TLS connection.

12

I have installed the datadog agent with the below command and configured otlp in datadog.yml

Can you please help me what I did wrong

datadogagent command:

DOCKER_CONTENT_TRUST=1 \
docker run --rm -d \
    --name datadog \
    -v /var/run/docker.sock:/var/run/docker.sock:ro \
    -v /proc/:/host/proc/:ro \
    -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
    -e DD_SITE="datadoghq.com" \
    -e DD_API_KEY=<YOUR_DATADOG_API_KEY> \
    -e DD_DOGSTATSD_NON_LOCAL_TRAFFIC=1 \
    -p 8125:8125/udp \
    datadog/agent:latest

datadog.yml configuration

otlp_config:

  receiver:

    protocols:

      grpc:

        endpoint: 0.0.0.0:4317



      http:

        endpoint: 0.0.0.0:4318





otlp_config:

  logs:

    enabled: true
13

You are probably missing to export from the container the HTTP 4318 port. I don’t see it on your arguments.

14

@codebien
when i’m using the port 4318 also getting the same error

K6_OTEL_HTTP_EXPORTER_INSECURE=true K6_OTEL_EXPORT_INTERVAL=120s K6_OTEL_HTTP_EXPORTER_ENDPOINT= 111.11.0.1:4318 K6_OTEL_METRIC_PREFIX=k6_ k6 run --tag test-id=Perf123 -o experimental-opentelemetry scripts/test.js

I got this IP address by using below command (it refers to Datadog agent running IP address)

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' datadog
15

Are you running the agent on the same machine? Did you try using localhost:4318?

16

yeah, but no use getting the same error and I observed every time the console showing

output: opentelemetry (grpc, localhost:4317)

/\ |‾‾| /‾‾/ /‾‾/

/\ / \ | |/ / / /

/ / \ | ( / ‾‾\

/ \ | |\ \ | (‾) |

/ __________ \ || \ _/ .io

execution: local

script: scripts/test.js

output: opentelemetry (grpc, localhost:4317)

scenarios: (100.00%) 1 scenario, 5 max VUs, 4m30s max duration (incl. graceful stop):

17
  1. You configured k6 to use the gRPC endpoint as the output reports. You have to use the HTTP endpoint for your case.
  2. I see you have a space
    K6_OTEL_HTTP_EXPORTER_ENDPOINT=<HERE>111.11.0.1:4318 in the previous posted option.
  3. Did you export the HTTP port as I recommended before?
18

where should I configure to use the HTTP endpoint in k6 instead of gRPC?

  • You configured k6 to use the gRPC endpoint as the output reports. You have to use the HTTP endpoint for your case.

It’s a typo, but I executed the command without space getting the same error

I see you have a space
K6_OTEL_HTTP_EXPORTER_ENDPOINT=<HERE>111.11.0.1:4318 in the previous posted option.

I have used the port number that was suggested. even though getting the same issue

Did you export the HTTP port as I recommended before?

19

You have to pass K6_OTEL_HTTP_EXPORTER_ENDPOINT and not K6_OTEL_GRPC_EXPORTER_ENDPOINT options.

What do you mean by used? You have to export the port on the container which means adding -p 4318:4318 on your run command for the container.

20

can you please confirm if is this the command I have to execute

K6_OTEL_HTTP_EXPORTER_INSECURE=true K6_OTEL_EXPORT_INTERVAL=120s K6_OTEL_HTTP_EXPORTER_ENDPOINT=111.11.0.1:4318 -p 4318:4318 K6_OTEL_METRIC_PREFIX=k6_ k6 run --tag test-id=Perf123 -o experimental-opentelemetry scripts/test.js

21

No, this isn’t what I expect. This should be the correct format:

K6_OTEL_HTTP_EXPORTER_INSECURE=true K6_OTEL_HTTP_EXPORTER_ENDPOINT=localhost:4318 K6_OTEL_METRIC_PREFIX=k6_ k6 run --tag test-id=Perf123 -o experimental-opentelemetry ./scripts/test.js

Then I expect you add the -p 4318:4318 to your docker run command.