Hi, so I’m running some tests and visualizing the results in Grafana via Prometheus remote write.
I’m often, if not always, seeing results like this:
So the response time seems to increase when the test is about to end. I don’t really understand how/why that could be. I’m thinking there could be some bug with how the calculations are done in Grafana. And it’s not like my server gets saturated after x time, the same thing seems to happen if I shorten or lengthen my test run. The response time just increases at the end.
My Grafana query for creating the avg response time lines is:
(histogram_sum(rate($metrics{testid=~"$testid", test="true"}[$__rate_interval]))
/
histogram_count(rate($metrics{testid=~"$testid", test="true"}[$__rate_interval])))
Which is a straight copy paste from the dashboard that’s linked in k6 prometheus remote write docs. (Except I have added test="true", I tag requests which are actually related to a test-run with that tag)
I’m thinking if those spikes could be related to my setup and teardown things. Because I have one slower request in each of them. But those requests are not tagged “test=true”, so they shouldn’t be taken into account?
Does anyone have any idea what could be going on here? Or how could I test whether the issue is with the Grafana calculation vs. those are the actual response times reported by k6.