Percentiles calculation on New Relic

Grafana k6 OSS Support
1

I’m sending metrics to New Relic using statsd (newrelic/nri-statsd:latest).
I have been trying to figure out how to calculate percentiles on New Relic over metric k6.http_req_duration without success, and requiring our help.

Following the documentation found here, on the image supplied, there are two widgets showing P90 values, but looking at the sample queries, we don’t have the p90 along the average/max/median query, and I can’t see how to take value from the sum of the durations…

SELECT sum(k6.http_req_duration.sum.percentiles) AS '90th' FROM Metric WHERE percentile = 90 TIMESERIES

How can I calculate P90 over K6 Metrics in New Relic, without using the pre-aggregated values? Can you provide an example?

I was looking into something like this, but whishing I could access the max value instead of the average:

select percentile(k6.http_req_duration.mean,90) AS 'ms' FROM Metric since 1800 seconds ago

Thanks

2

Hello @lssoares !
I was checking your question and it seems to be slightly more related to NewRelic than k6, but I think where I could give you some insights from the k6 side.
k6 doesn’t send percentile values as metrics per se .
This was discussed in this thread: Add 99, 99.5 percentile stats to built-in HTTP trend metrics - OSS Support - Grafana Labs Community Forums
Most probably, New Relic is receiving each response time as a single metric.
I was doing some research and it seems like it is indeed a matter with NewRelic’s query.
Found there were some changes with the percentile function. Improvements to NRQL percentile() | New Relic Documentation

I found a couple examples for max and for 90pct in k6 documentation. Pasting them below, but again it seems to me more like a NR query matter.
For 90th
SELECT sum(k6.http_req_duration.sum.percentiles) AS '90th' FROM Metric WHERE percentile = 90 TIMESERIES
For Max
SELECT max(k6.http_req_duration.summary) AS 'Max Duration' FROM Metric TIMESERIES

Hope that helped. If not, let me know as it has been a while since I last played with NR, and it would be an interesting finding to have.
Best of lucks!
Gracias,
Leandro Melendez

1 Like
3

Hi @SrPerf :slight_smile:
Thank you for your help.

I agree that I might have to ask for percentiles calculations on NewRelic forum, but I would like to understand first the examples provided by K6.

So, for a particular test, I get this result:

On K6 summary:
http_req_duration...............: min=15.6ms med=25.78ms avg=37.07ms max=4.81s p(90)=57.48ms p(95)=92.41ms p(98)=142.98ms p(99.999)=4.15s

On new Relic (using the queries provided by k6 documentation):

image
image625×611 27 KB

  • Max, med and avg seems OK when compared to K6 results.
  • About P90, I have a lot of doubts:

  1. Regarding the 800 000 as a peak seen in the graph…what is the measurement unit? Miliseconds? How is it meaningful?

  2. Changing the query for other percentiles (ex: where percentile = 95), would give me no results. How can I see those?

Thanks,
Leandro

4

Hey @lssoares !
Wow, seems like you and I have the same name, That is really cool!

For the NewRelic, it seems like it is getting the right data points as it is correctly displaying the max, med and avg.

On on the percentiles, it really seems weird.
It seems to me like it has to do with the calculation on New Relic. Not so much on the k6 side.
Unfortunately I do not have a NR environment to test and tune that. I got a temporary free access but seems to not to be allowing me to send the data. Need to work that out.
I will check with the k6 team, who wrote that part of the documentation, probably will give better insights. But seems worth touching base with NewRelic.

On the other hand, I would first write the query to list all the data points and compare with k6’s to make sure you have them all in NR.
After that is just figuring out the percentile functions.

Please let me know if it received them all and I will keep you posted if I find out something.
Gracias,
Leandro

1 Like
5

Hey @SrPerf thanks for the detailed replies in this thread.
I was wondering if you had the chance to look into this? I’m building a proof of concept for my company and would like to see these metrics being pushed to NR correctly. I’m blocked by the same issue, I can’t seem to get percentiles other than the 90th.
Cheers

6

Hey @msc
Sorry I was on vacation and could not get a chance to look into this… I will try to find some time soon and hopefully I will be able to pin down the problem.
QQ. Are you experiencing the exact same situation?
Gracias,
Leandro

7

Hey Leandro. Not usual to see someone sharing the first name with me :slight_smile:

I would expect to see results correctly in the example provided by k6, but anyway, I would be glad to contribute.
Can you please provide the step to enable logging of the metrics being sent to new relic on the k6 agent, so I can compare with what I have in new relic? Give me some directions here…

Thank you.

8

Hi Team,
We are facing the exact same issue, appreciate if we can get some inputs on solving this.

@SrPerf can you also please elaborate "write the query to list all the data points "? I can try that out. I tried below but it does not work.

SELECT percentile(k6.http_req_duration, 90, 95, 99) FROM Metric