K6 x Datadog Issues

[kghelani]

I am trying to output my k6 results to datadog locally.

I have tried the following:

1. Start datadog

DOCKER_CONTENT_TRUST=1 \
docker run --rm -d \
    --name datadog \
    -v /var/run/docker.sock:/var/run/docker.sock:ro \
    -v /proc/:/host/proc/:ro \
    -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
    -e DD_SITE="datadoghq.com" \
    -e DD_API_KEY=<YOUR_DATADOG_API_KEY> \
    -e DD_DOGSTATSD_NON_LOCAL_TRAFFIC=1 \
    -p 8125:8125/udp \
    datadog/agent:latest

2. Run k6

K6_STATSD_ENABLE_TAGS=true k6 run --out statsd <my test>.js

I have also tried while supplying K6_STATSD_ADDR with

• datadog:8125
• the IP from my datadog docker process using
  docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' datadog

With all these, I get a similar error from k6 that the metrics are not going though.
ERRO[0005] Couldn't flush a batch error="write udp 192.168.1.246:49185->172.17.0.2:8125: write: connection refused" output=statsd

Any ideas would be appreciated, thanks!

[olegbespalov]

Hi @kghelani

The issue seems not really related to the k6. It just tells you that couldn’t reach the address.

What I see is that you exposing the port to the host:

@kghelani

-p 8125:8125/udp

So in theory it should work with the standard k6’s K6_STATSD_ADDR which is localhost:8125. For sure if k6 is running on the same host machine.

If it doesn’t work with this address I’d recommend checking the documentation of the datadog/agent to see why it could refuse the connection.

Hope that helps,
Cheers!

[gurgelv]

I encountered the same issue as @kghelani (write: connection refused" output=statsd), and seemed to overcome it by using the container IP directly, as such:

$ docker inspect -f ‘{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}’ dd-agent
172.17.0.2

Then:

K6_STATSD_ADDR=172.17.0.2:8125 K6_STATSD_ENABLE_TAGS=true k6 run --out statsd ./load/sample.test.js

No metrics seem to have been received by the agent, although the test seems to run successfully:

Here’s how I initialized my local DD Agent FYI:

DOCKER_CONTENT_TRUST=1
docker run --rm -d
–name dd-agent
-v /var/run/docker.sock:/var/run/docker.sock:ro
-v /proc/:/host/proc/:ro
-v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
-e DD_SITE=“datadoghq.com”
-e DD_API_KEY=“”
-e DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true
-e NON_LOCAL_TRAFFIC=true
-p 8125:8125/udp
datadog/agent:latest

Any help appreciated!

[rodyb]

Hi @kghelani @gurgelv

Do I need subscription of k6 to show tags in data dog? I am running script in...

Let me know if this works for you.

[ghassenjn]

Hi @rodyb,
I’ve done almost the same, Still no metrics forwarded to Datadog, without errors,
I don’t see what the problem can be,

below the nc command return to be sure that the connection is well established with to agent through this IP address,

nc -vz -u 172.17.0.2 8125
Connection to 172.17.0.2 port 8125 [udp/*] succeeded!

Any other suggestions ?

Thank you,
Ghassen.

[rodyb]

hi @ghassenjn,

Can you try this and let me know what the error shows?

version: '3.7'
volumes:
  datadog:

services:
  datadog:
    image: gcr.io/datadoghq/agent:7
    networks:
      - k6
    container_name: dd-agent
    restart: always
    ports:
      - 8125:8125
    environment:
      - DD_API_KEY=<KEY>
      - DD_SITE=datadoghq.com 
      - DD_DOGSTATSD_NON_LOCAL_TRAFFIC=1
      - DD_APM_ENABLED=true
      - DD_LOGS_ENABLED=true
      - DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true
      - DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro

  k6:
    image: loadimpact/k6
    networks:
      - k6
    container_name: k6
    depends_on:
      datadog:
        condition: service_healthy
    command: run -e K6_STATSD_ADDR=datadog:8125 -e K6_STATSD_ENABLE_TAGS=true --out statsd tests/<Sometest.js>
    volumes:
      - .:/app:delegated

networks:
  k6:

I am not sure what problem you exactly have, however I also ran into an issue with this

 DD_SITE=datadoghq.com 

Make sure this is either europe or US, depending on where Datadog is hosted for you.

Can you show me the errors that you see?

[ghassenjn]

Hi @rodyb,

Thank you for your response,
I got those errors =>

dd-agent  | 2023-09-05 11:48:27 UTC | CORE | ERROR | (comp/forwarder/defaultforwarder/worker.go:198 in process) | Error while processing transaction: error while sending transaction, rescheduling it: Post "https://7-47-0-app.agent.datadoghq.eu/api/v1/check_run": tls: failed to verify certificate: x509: certificate signed by unknown authority

2023-09-05 11:48:38 UTC | PROCESS | ERROR | (comp/forwarder/defaultforwarder/worker.go:194 in process) | Too many errors for endpoint 'https://process.datadoghq.eu/api/v1/container': retrying later```

[rodyb]

Hi @ghassenjn,

I don’t know what site you are testing but can you try to run it with this flag?

k6 run tests/script.js --insecure-skip-tls-verify 

This should disable the certificate.

X509: certificate signed by unknown authority

I have started trying out k6 today and I am impressed. I see a problem with https requests made to a test server. WARN[0003] Request Failed error=“Get "https://127.0.0.1:4433": x509: certificate signed by unknown authority” it is self signed certificate. I don’t want disable the tls verify. Because we are testing tls 1.3 testing please help me with this guys Thanks in advance

[ghassenjn]

Sorry, I update my post, I got the same problem which is related to the tls certificate,

dd-agent  | 2023-09-05 14:40:53 UTC | CORE | ERROR | (comp/forwarder/defaultforwarder/transaction/transaction.go:115 in 4) | TLS Handshake failure: tls: failed to verify certificate: x509: certificate signed by unknown authority

[rodyb]

Hi @ghassenjn,

I need a bit more information to understand the issue better.

A couple of things I need.

• How does your test look like ? Can you share that here ?
• Can you share here, how you run the test? (what command exactly?)
• Have you tried to use a different URL, then you use now and see if the problem persists?

Thanks.