I’m running some tests using k6 with xk6 build of xk6-output-prometheus-remote sending the data to Grafana Cloud’s Prometheus data store.
My issue is that the metrics being sent appear to be pre-aggregated - that is, what I have available is avg/min/max/90/95 - not individual request metrics, e.g. for duration:
When I’ve done local tests using InfluxDB as the backend to Grafana I get the data for each individual request which allows me to do the analysis/aggregation using the tools available but when sending to Prometheus I lose that ability.
Is there a way of changing the Prometheus metrics represent the individual requests rather than an aggregate of sorts?
Secondary to that, is there a way of sending data from multiple running k6 tests in such a way that doesn’t cause an ‘out of order sample’ error, via tagging or something. I’ll admit my knowledge of Prometheus is fairly limited - I just want to be able to keep track of some short term k6 test data.
Welcome to the forum! You’re right, by default PRW extension is aggregating metrics. This can be switched off with K6_PROMETHEUS_MAPPING=raw option: then extension will be sending values as is, without aggregation. Prometheus then would treat all of them as Gauges and none of the metrics would get suffixes like _max or _p95.
is there a way of sending data from multiple running k6 tests in such a way that doesn’t cause an ‘out of order sample’ error, via tagging or something.
Could you elaborate a bit on what do you mean by “multiple running k6 tests”? Does it mean that you’re executing several test runs at the same time that are all writing to the same Prometheus? The errors out of order sample happen because Prometheus expects to receive metric samples in order. If you are indeed running several k6 runs simultaneously then it might make sense to start distinguishing metrics by some additional tag, like k6_instance_id=X or, if possible, run them sequentially.
That’s absolutely what I was looking for, thank you hugely.
As for elaboration on the second question, I think you’ve already figured it out and that sounds like you’ve given me a solution, I’ll give it a test and will see if the issue is sorted.
Either way the first issue was my primary problem so thank you for sorting it.
