Test Results from Official k6 Grafana dashboard and on k6 console are mismatched

Grafana k6 OSS Support
1

Hi,

I was trying to push k6 test results to be viewed in Grafana with Prometheus-remote-write (along with Native Histograms enabled)

I have also imported the official k6 dashboard in Grafana to view the metrics.

I make use of ‘Groups’ in my tests, and modified the dashboard slightly to view the response times w.r.t. groups.

But observed that P50, P95 response times displayed in the grafana do not match with the response times I see in the k6 result console.

Results on k6 console for group ListInsts P50 is 1.27s

image
image1093×110 5.99 KB

same result in grafana k6 dashboard is 2.18s

image
image1920×903 61.4 KB

This is the query used:
histogram_quantile(0.50, sum by(group) (rate($metrics{group=~“$group”}[$__rate_interval])))

Could you please help with resolving this discrepancy.

Thanks,
Akshayini

3

Could someone please tell?

4

Hi @Akshayini :wave:

At first glance, my intuition would be that the query you use in Prometheus is based on the group=~ PromQL matcher. The Prometheus documentation states: “=~: Select labels that regex-match the provided string.”

Could the expression match all groups starting with ListInsts in this specific request? As you sum by group, that would result in a value consisting of the sum of the median values of the ListInsts_filtered and ListInsts. Which from the number you display takes us to 1270 + 875, which gives us 2145 milliseconds. Close enough to the number displayed that I wonder if this is what might be happening.

only If you’ve established that’s not it, could you confirm the following:

  • Do you see a difference in results consistently (across many test runs), or is it punctual and maybe even sporadic?
  • Do you observe the same phenomenon for other metrics and groups?
  • Could you share with us a script that reproduces the problem?
  • Do the max and min values returned in the summary match what you observe in Prometheus?

Let me know if that was helpful :bowing_man: