Forward stats from multiple k6 workers to single Datadog agent

Grafana k6 Converters & Integrations
1

My k6 load test runs on multiple Kubernetes pods (to simulate heavy load). It works fine with K6_OUT=influxdb, but when “datadog” out is used, there are issues with aggregating data.

Specifically, I can see that counters (vus, vus_max, http_reqs etc…) are not aggregated from multiple sources (k6 worker pods) and only single pod metrics are displayed. Datadog dashboard is set to “sum” counters, but it looks like the problem is in datadog-agent/k6 integration.

Current k8s design is following:

  • single datadog-agent pod that exposed as a service
  • multiple k6 workers (e.g. 30 instances) that have K6_OUT=datadog, K6_DATADOG_ADDR=[cluster-dns-name]]

Any ideas what’s wrong here?

I suspect that the problem can be fixed if each k6 worker will forward stats to individual datadog-agent (per node), but that increases complexity and Datadog will charge more (since pricing it based on number of agents).

2

Hi @romitch,

I think you will need to use --tag instance=X where X is different between all your k6 instances in order for datadog to be able to differentiate them. You can obviously use a different tag name, but without this … datadog is getting basically a counter vus (for example) with the exact same tags and it just can’t know that they are coming from multiple instances.

Hope this helps

1 Like
3

Hi @mstoykov, thanks for the advice! the problem is fixed now using --tag instance=X