Wrong percentile values for metrics shown in Datadog

Naveen · February 23, 2024, 10:20am

Hi Team, Any update on these issues

Wrong percentile values for metrics shown in Datadog

opened 06:33PM - 13 Dec 22 UTC

bug evaluation needed

### Brief summary As reported several times on the forum ([1](https://communi…ty.k6.io/t/trend-metrics-on-datadog-percentile/3995), [2](https://community.k6.io/t/http-req-duration-wrong-datadog/440/), [3](https://community.k6.io/t/why-do-k6-http-req-duration-max-and-k6-http-req-duration-95percentile-have-the-same-value-in-datadog/5487/3)), the percentile values users can visualize in Datadog are often wrong when compared to the end-of-test summary shown by k6. As explained [here](https://community.k6.io/t/http-req-duration-wrong-datadog/440/5), this is likely caused by the additional aggregation done by the DogStatsD agent, which by default only generates the `95percentile` metric, which is then aggregated again by the query used in the Datadog graph. Besides resulting in wrong values, users are unable to use raw metric data to generate any percentile value they need. ### k6 version 0.41.0 ### OS any ### Steps to reproduce the problem 1. Follow the steps to setup the Datadog Agent, and run a test as explained [here](https://k6.io/docs/results-output/real-time/datadog/). 2. Go to the Datadog web UI and visualize the `k6.http_req_duration.max` and `k6.http_req_duration.95percentile` metrics, and compare them to the end-of-test summary shown by k6. Notice that they don't match. ### Expected behaviour - The percentiles shown in Datadog should match the end-of-test summary shown by k6. - The user should be able to generate any percentile over raw metric data sent to Datadog. Ideally, the Datadog Agent (DogStatsD) shouldn't do any aggregation at all. ### Actual behaviour - The percentiles are different, and the Datadog `95percentile` is confusing. - The user can only use a limited number of metrics, and most pre-aggregated ones show wrong values. ### Suggested solution After going through the Datadog documentation, it seems it's possible to send raw data that won't be aggregated by the Datadog Agent using the distribution metric type ([1](https://docs.datadoghq.com/metrics/distributions/), [2](https://docs.datadoghq.com/metrics/types/?tab=distribution#metric-types)). This would not only avoid the aggregation, but allow users to generate any percentile value they need over the raw data. The drawback would possibly be overloading the ingest pipeline (either of the Datadog Agent or of Datadog itself, hitting some API limits, etc.), so this needs to be thoroughly tested. [Currently we send `Count`, `Gauge` and other metric types that will be aggregated](https://github.com/grafana/k6/blob/a2ab89abfdc8c12fa28b3d25092b2a1d9310f8c4/output/statsd/output.go#L52-L71), but the `datadog-go/statsd` client we use [also supports the `Distribution` metric](https://github.com/DataDog/datadog-go/blob/6391269270019b7fa3888a91b854a4a00d6772bd/statsd/statsd.go#L197-L198). In addition to evaluating whether this change works for some of our metric types, we should also ensure that we don't break support for any other StatsD backends.

github.com/grafana/k6

Future of StatsD outputs

opened 03:30PM - 21 Mar 23 UTC

imiric

enhancement

The k6-core team recently had an internal meeting to discuss the current status …of StatsD outputs in k6, the several issues reported by users, and the best way to move forward that benefits both users, and maintenance of k6. ## Background The [`statsd` output](https://k6.io/docs/results-output/real-time/statsd/) is a generic implementation of the [StatsD line protocol](https://github.com/statsd/statsd#usage). It allows pushing metrics to any StatsD server, and, in theory, any StatsD backend. Under the hood, it uses the [DogStatsD client library](https://github.com/DataDog/datadog-go), but it doesn't use any Datadog-specific functionality. This is why the same output can be used to send metrics to both [Datadog](https://k6.io/docs/results-output/real-time/datadog/) and [New Relic](https://k6.io/docs/results-output/real-time/new-relic/). ## Issues The problem is that, on its own, this generic implementation is very basic, and only supports [3 metric types](https://github.com/statsd/statsd/blob/master/docs/metric_types.md) (Counters, Gauges, and Sets). Datadog and New Relic have extended the protocol to add support for more metric types, though these changes, are not standard, and likely wouldn't work across different servers and backends. So we can't use this functionality in the `statsd` output without risking breaking generic StatsD support, or specifically supporting individual backends, which we want to avoid. There are several issues reported by users on the forum: - https://community.k6.io/t/trend-metrics-on-datadog-percentile/3995 - https://community.k6.io/t/http-req-duration-wrong-datadog/440 - https://community.k6.io/t/why-do-k6-http-req-duration-max-and-k6-http-req-duration-95percentile-have-the-same-value-in-datadog/5487 - https://community.k6.io/t/percentiles-calculation-on-new-relic/3690 - https://community.k6.io/t/how-to-get-the-correct-values-for-p-90-on-k6-newrelic/5192 Some of these, particularly those related to percentile values, and summarized in #2819, could be fixed by using the non-standard metric types, such as [Distribution](https://docs.datadoghq.com/metrics/distributions/). But, as mentioned above, we can't use it directly in the output. ## Proposed solutions We agreed that the way forward consists of several steps: 1. Research the status of alternative protocol support for Datadog and New Relic. For example, [New Relic supports Prometheus Remote Write natively](https://docs.newrelic.com/docs/infrastructure/prometheus-integrations/get-started/send-prometheus-metric-data-new-relic/#remote-write), which k6 can output via [xk6-output-prometheus-remote](https://github.com/grafana/xk6-output-prometheus-remote). Datadog only supports [pulling metrics from Prometheus endpoints](https://docs.datadoghq.com/containers/kubernetes/prometheus/) (not Remote Write), but could technically support it via [Vector](https://github.com/vectordotdev/vector) (owned by Datadog), which does have a [RW source](https://vector.dev/docs/reference/configuration/sources/prometheus_remote_write/). 2. Depending on the results of the above research, create separate k6 Datadog and New Relic output extensions. These will initially not solve any of the reported issues, and will essentially be an extraction and duplication of the current `statsd` output. The benefit of this is that being separate outputs would allow them to use any backend-specific functionality. The k6-core team will be in charge of this development and initial maintenance. The reason this depends on the research from step 1, is because if we find that both backends work fine with Prometheus RW, then we can just update our documentation instead of creating separate extensions. Or if only New Relic does, then only create the extension for Datadog, etc. 3. Contact the respective Datadog and New Relic teams, explain our situation and existing issues, and politely transfer maintenance of the extensions to them. We believe their teams will have better knowledge about the best ways to fix the current issues, and would be better equipped to continue maintaining the integration with their backends. We would essentially like to achieve the same situation as with the current [Elasticsearch output](https://github.com/elastic/xk6-output-elasticsearch), which is maintained by Elastic. 4. Decide whether to continue supporting the plain StatsD output in k6, or if it should be deprecated in favor of Prometheus or [OpenTelemetry](https://github.com/grafana/k6/issues/2557). [The StatsD project](https://github.com/statsd/statsd) itself is quite old, and hasn't seen updates in years. With all its limitations, it's unlikely anyone is still using it. Especially since nowadays most backends support modern protocols, like Prometheus, OpenTelemetry or OpenMetrics. I can create sub-issues for each step if it makes more sense. Let me know.

Still, we didn’t get exact k6 result values in Datadog
Can you look into it?

codebien · March 11, 2024, 1:59pm

Hey @Naveen,
the status regarding Statsd output is tracked in the issues you’ve linked. There isn’t any update at the moment.

Currently, the priority for us regarding metrics output is to add support for OpenTelemetry as it supports several metrics backends.

I hope it helps.