Naveen
February 23, 2024, 10:20am
1
Hi Team, Any update on these issues
opened 06:33PM - 13 Dec 22 UTC
bug
evaluation needed
### Brief summary
As reported several times on the forum ([1](https://communi… ty.k6.io/t/trend-metrics-on-datadog-percentile/3995), [2](https://community.k6.io/t/http-req-duration-wrong-datadog/440/), [3](https://community.k6.io/t/why-do-k6-http-req-duration-max-and-k6-http-req-duration-95percentile-have-the-same-value-in-datadog/5487/3)), the percentile values users can visualize in Datadog are often wrong when compared to the end-of-test summary shown by k6.
As explained [here](https://community.k6.io/t/http-req-duration-wrong-datadog/440/5), this is likely caused by the additional aggregation done by the DogStatsD agent, which by default only generates the `95percentile` metric, which is then aggregated again by the query used in the Datadog graph.
Besides resulting in wrong values, users are unable to use raw metric data to generate any percentile value they need.
### k6 version
0.41.0
### OS
any
### Steps to reproduce the problem
1. Follow the steps to setup the Datadog Agent, and run a test as explained [here](https://k6.io/docs/results-output/real-time/datadog/).
2. Go to the Datadog web UI and visualize the `k6.http_req_duration.max` and `k6.http_req_duration.95percentile` metrics, and compare them to the end-of-test summary shown by k6. Notice that they don't match.
### Expected behaviour
- The percentiles shown in Datadog should match the end-of-test summary shown by k6.
- The user should be able to generate any percentile over raw metric data sent to Datadog. Ideally, the Datadog Agent (DogStatsD) shouldn't do any aggregation at all.
### Actual behaviour
- The percentiles are different, and the Datadog `95percentile` is confusing.
- The user can only use a limited number of metrics, and most pre-aggregated ones show wrong values.
### Suggested solution
After going through the Datadog documentation, it seems it's possible to send raw data that won't be aggregated by the Datadog Agent using the distribution metric type ([1](https://docs.datadoghq.com/metrics/distributions/), [2](https://docs.datadoghq.com/metrics/types/?tab=distribution#metric-types)). This would not only avoid the aggregation, but allow users to generate any percentile value they need over the raw data.
The drawback would possibly be overloading the ingest pipeline (either of the Datadog Agent or of Datadog itself, hitting some API limits, etc.), so this needs to be thoroughly tested.
[Currently we send `Count`, `Gauge` and other metric types that will be aggregated](https://github.com/grafana/k6/blob/a2ab89abfdc8c12fa28b3d25092b2a1d9310f8c4/output/statsd/output.go#L52-L71), but the `datadog-go/statsd` client we use [also supports the `Distribution` metric](https://github.com/DataDog/datadog-go/blob/6391269270019b7fa3888a91b854a4a00d6772bd/statsd/statsd.go#L197-L198).
In addition to evaluating whether this change works for some of our metric types, we should also ensure that we don't break support for any other StatsD backends.
opened 03:30PM - 21 Mar 23 UTC
enhancement
The k6-core team recently had an internal meeting to discuss the current status … of StatsD outputs in k6, the several issues reported by users, and the best way to move forward that benefits both users, and maintenance of k6.
## Background
The [`statsd` output](https://k6.io/docs/results-output/real-time/statsd/) is a generic implementation of the [StatsD line protocol](https://github.com/statsd/statsd#usage). It allows pushing metrics to any StatsD server, and, in theory, any StatsD backend.
Under the hood, it uses the [DogStatsD client library](https://github.com/DataDog/datadog-go), but it doesn't use any Datadog-specific functionality. This is why the same output can be used to send metrics to both [Datadog](https://k6.io/docs/results-output/real-time/datadog/) and [New Relic](https://k6.io/docs/results-output/real-time/new-relic/).
## Issues
The problem is that, on its own, this generic implementation is very basic, and only supports [3 metric types](https://github.com/statsd/statsd/blob/master/docs/metric_types.md) (Counters, Gauges, and Sets). Datadog and New Relic have extended the protocol to add support for more metric types, though these changes, are not standard, and likely wouldn't work across different servers and backends. So we can't use this functionality in the `statsd` output without risking breaking generic StatsD support, or specifically supporting individual backends, which we want to avoid.
There are several issues reported by users on the forum:
- https://community.k6.io/t/trend-metrics-on-datadog-percentile/3995
- https://community.k6.io/t/http-req-duration-wrong-datadog/440
- https://community.k6.io/t/why-do-k6-http-req-duration-max-and-k6-http-req-duration-95percentile-have-the-same-value-in-datadog/5487
- https://community.k6.io/t/percentiles-calculation-on-new-relic/3690
- https://community.k6.io/t/how-to-get-the-correct-values-for-p-90-on-k6-newrelic/5192
Some of these, particularly those related to percentile values, and summarized in #2819, could be fixed by using the non-standard metric types, such as [Distribution](https://docs.datadoghq.com/metrics/distributions/). But, as mentioned above, we can't use it directly in the output.
## Proposed solutions
We agreed that the way forward consists of several steps:
1. Research the status of alternative protocol support for Datadog and New Relic.
For example, [New Relic supports Prometheus Remote Write natively](https://docs.newrelic.com/docs/infrastructure/prometheus-integrations/get-started/send-prometheus-metric-data-new-relic/#remote-write), which k6 can output via [xk6-output-prometheus-remote](https://github.com/grafana/xk6-output-prometheus-remote).
Datadog only supports [pulling metrics from Prometheus endpoints](https://docs.datadoghq.com/containers/kubernetes/prometheus/) (not Remote Write), but could technically support it via [Vector](https://github.com/vectordotdev/vector) (owned by Datadog), which does have a [RW source](https://vector.dev/docs/reference/configuration/sources/prometheus_remote_write/).
2. Depending on the results of the above research, create separate k6 Datadog and New Relic output extensions. These will initially not solve any of the reported issues, and will essentially be an extraction and duplication of the current `statsd` output. The benefit of this is that being separate outputs would allow them to use any backend-specific functionality.
The k6-core team will be in charge of this development and initial maintenance.
The reason this depends on the research from step 1, is because if we find that both backends work fine with Prometheus RW, then we can just update our documentation instead of creating separate extensions. Or if only New Relic does, then only create the extension for Datadog, etc.
3. Contact the respective Datadog and New Relic teams, explain our situation and existing issues, and politely transfer maintenance of the extensions to them.
We believe their teams will have better knowledge about the best ways to fix the current issues, and would be better equipped to continue maintaining the integration with their backends.
We would essentially like to achieve the same situation as with the current [Elasticsearch output](https://github.com/elastic/xk6-output-elasticsearch), which is maintained by Elastic.
4. Decide whether to continue supporting the plain StatsD output in k6, or if it should be deprecated in favor of Prometheus or [OpenTelemetry](https://github.com/grafana/k6/issues/2557).
[The StatsD project](https://github.com/statsd/statsd) itself is quite old, and hasn't seen updates in years. With all its limitations, it's unlikely anyone is still using it. Especially since nowadays most backends support modern protocols, like Prometheus, OpenTelemetry or OpenMetrics.
I can create sub-issues for each step if it makes more sense. Let me know.
Still, we didn’t get exact k6 result values in Datadog
Hey @Naveen ,
Currently, the priority for us regarding metrics output is to add support for OpenTelemetry as it supports several metrics backends.
I hope it helps.
Naveen
June 25, 2024, 11:19am
5
can we expect that issue to be resolved in 2025 Q1
Hi @codebien while running a load test I’m getting the following errors in the console and I couldn’t able to see the data in datadog
WARN[0003] Couldn’t send 4 out of 35 metrics. Enable verbose logging with --verbose to see individual errors output=statsd
Note: Fyi…, I have installed k6 and datadog agent successfully
Naveen
July 7, 2024, 12:29pm
8
Issue resolved by following this document and command
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' datadog```
https://github.com/grafana/k6-operator/issues/63#:~:text=echo%20%22%3A%3Aset%2Doutput%20name%3DddIP%3A%3A%24(%20docker%20inspect%20%2Df%20%27%7B%7Brange%20.NetworkSettings.Networks%7D%7D%7B%7B.IPAddress%7D%7D%7B%7Bend%7D%7D%27%20datadog%20)%22
Naveen
August 16, 2024, 2:33am
9
HI @codebien ,
Based on the [document](https:// Statsd seems to be dropping metrics probably because of UDP · Issue #2044 · grafana/k6 · GitHub) I have updated k6 version to v0.53.0. then I executed a script but I’m unable to export the results to Datadog
This is the following command I tried and got the below errorcommand:
K6_OTEL_HTTP_EXPORTER_ENDPOINT=http://localhost:4318 K6_OTEL_METRIC_PREFIX=k6_ k6 run --tag test-id=Perf123 -o experimental-opentelemetry scripts/loadtest.js
error message
failed to upload metrics: context deadline exceeded: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:4317: connect: connection refused"
Naveen
August 19, 2024, 6:17pm
10
HI @codebien ,
Based on the [document](https:// Statsd seems to be dropping metrics probably because of UDP · Issue #2044 · grafana/k6 · GitHub) I have updated k6 version to v0.53.0. then I executed a script but I’m unable to export the results to Datadog
This is the following command I tried and got the below errorcommand:
K6_OTEL_HTTP_EXPORTER_ENDPOINT=http://localhost:4318 K6_OTEL_METRIC_PREFIX=k6_ k6 run --tag test-id=Perf123 -o experimental-opentelemetry scripts/loadtest.js
error message
failed to upload metrics: context deadline exceeded: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:4317: connect: connection refused"
Hey @Naveen ,
Thanks
Naveen
August 26, 2024, 2:49pm
13
