Response time increases at the end of test

Hi, so I’m running some tests and visualizing the results in Grafana via Prometheus remote write.

I’m often, if not always, seeing results like this:

So the response time seems to increase when the test is about to end. I don’t really understand how/why that could be. I’m thinking there could be some bug with how the calculations are done in Grafana. And it’s not like my server gets saturated after x time, the same thing seems to happen if I shorten or lengthen my test run. The response time just increases at the end.

My Grafana query for creating the avg response time lines is:

(histogram_sum(rate($metrics{testid=~"$testid", test="true"}[$__rate_interval]))
histogram_count(rate($metrics{testid=~"$testid", test="true"}[$__rate_interval])))

Which is a straight copy paste from the dashboard that’s linked in k6 prometheus remote write docs. (Except I have added test="true", I tag requests which are actually related to a test-run with that tag)

I’m thinking if those spikes could be related to my setup and teardown things. Because I have one slower request in each of them. But those requests are not tagged “test=true”, so they shouldn’t be taken into account?

Does anyone have any idea what could be going on here? Or how could I test whether the issue is with the Grafana calculation vs. those are the actual response times reported by k6.

Hi @amogus,

Can you share what k6 version you are using (e.g. running k6 version) and a (sanitized) version of your script? It will help if we can have a look at the test.