Hey folks, I’ve been playing around with the experimental-prometheus-rw feature and in my case I only see a subset of the k6 OSS metrics reported in the Grafana metrics viewer…
any thoughts on how I can push the remaining default k6_* metrics?
Hi @PlayStay
From the screenshot it seems you are sending the metrics to Cortex. Can you share the version? I would test this locally to see if I can reproduce the scenario. I’ve tested the prometheus write output with InfluxDb and it works, though there might be some incompatibilities with Cortex we can investigate.
Could you query directly your Cortex to get thet list of k6_* metrics? Are your results the same you see on Grafana when you filter for k6_? That is, you see reduced list of metrics?
Something similar to (localhost with no authentication):
curl -g 'http://localhost:9090/api/v1/series?' --data-urlencode 'match[]=up' --data-urlencode 'match[]={__name__=~".*k6_.*"}'
In my lab (local Prometheus v2.40.7 after running the es6sample.js test) I am getting:
{
"status": "success",
"data": [
{
"__name__": "k6_checks_rate",
"check": "X-Test header is correct",
"group": "::my group::json",
"scenario": "default"
},
{
"__name__": "k6_checks_rate",
"check": "always fails",
"group": "::my group::nested",
"scenario": "default"
},
{
"__name__": "k6_checks_rate",
"check": "always passes",
"group": "::my group::nested",
"scenario": "default"
},
{
"__name__": "k6_checks_rate",
"check": "content type is html",
"group": "::my group::html",
"scenario": "default"
},
{
"__name__": "k6_checks_rate",
"check": "random value is \u003c 0.5",
"group": "::my group",
"scenario": "default"
},
{
"__name__": "k6_checks_rate",
"check": "status is 200",
"group": "::my group::html",
"scenario": "default"
},
{
"__name__": "k6_checks_rate",
"check": "status is 200",
"group": "::my group::json",
"scenario": "default"
},
{
"__name__": "k6_checks_rate",
"check": "top-level test",
"scenario": "default"
},
{
"__name__": "k6_checks_rate",
"check": "welcome message is correct",
"group": "::my group::html",
"scenario": "default"
},
{
"__name__": "k6_data_received_total",
"scenario": "default"
},
{
"__name__": "k6_data_sent_total",
"scenario": "default"
},
{
"__name__": "k6_group_duration_p99",
"group": "::my group",
"scenario": "default"
},
{
"__name__": "k6_group_duration_p99",
"group": "::my group::html",
"scenario": "default"
},
{
"__name__": "k6_group_duration_p99",
"group": "::my group::json",
"scenario": "default"
},
{
"__name__": "k6_group_duration_p99",
"group": "::my group::nested",
"scenario": "default"
},
{
"__name__": "k6_http_req_blocked_p99",
"expected_response": "true",
"group": "::my group::html",
"method": "GET",
"name": "http://test.k6.io/",
"proto": "HTTP/1.1",
"scenario": "default",
"status": "308",
"url": "http://test.k6.io/"
},
{
"__name__": "k6_http_req_blocked_p99",
"expected_response": "true",
"group": "::my group::html",
"method": "GET",
"name": "https://test.k6.io/",
"proto": "HTTP/1.1",
"scenario": "default",
"status": "200",
"tls_version": "tls1.3",
"url": "https://test.k6.io/"
},
{
"__name__": "k6_http_req_blocked_p99",
"expected_response": "true",
"group": "::my group::json",
"method": "GET",
"name": "https://httpbin.org/get",
"proto": "HTTP/2.0",
"scenario": "default",
"status": "200",
"tls_version": "tls1.2",
"url": "https://httpbin.org/get"
},
{
"__name__": "k6_http_req_connecting_p99",
"expected_response": "true",
"group": "::my group::html",
"method": "GET",
"name": "http://test.k6.io/",
"proto": "HTTP/1.1",
"scenario": "default",
"status": "308",
"url": "http://test.k6.io/"
},
{
"__name__": "k6_http_req_connecting_p99",
"expected_response": "true",
"group": "::my group::html",
"method": "GET",
"name": "https://test.k6.io/",
"proto": "HTTP/1.1",
"scenario": "default",
"status": "200",
"tls_version": "tls1.3",
"url": "https://test.k6.io/"
},
{
"__name__": "k6_http_req_connecting_p99",
"expected_response": "true",
"group": "::my group::json",
"method": "GET",
"name": "https://httpbin.org/get",
"proto": "HTTP/2.0",
"scenario": "default",
"status": "200",
"tls_version": "tls1.2",
"url": "https://httpbin.org/get"
},
{
"__name__": "k6_http_req_duration_p99",
"expected_response": "true",
"group": "::my group::html",
"method": "GET",
"name": "http://test.k6.io/",
"proto": "HTTP/1.1",
"scenario": "default",
"status": "308",
"url": "http://test.k6.io/"
},
{
"__name__": "k6_http_req_duration_p99",
"expected_response": "true",
"group": "::my group::html",
"method": "GET",
"name": "https://test.k6.io/",
"proto": "HTTP/1.1",
"scenario": "default",
"status": "200",
"tls_version": "tls1.3",
"url": "https://test.k6.io/"
},
{
"__name__": "k6_http_req_duration_p99",
"expected_response": "true",
"group": "::my group::json",
"method": "GET",
"name": "https://httpbin.org/get",
"proto": "HTTP/2.0",
"scenario": "default",
"status": "200",
"tls_version": "tls1.2",
"url": "https://httpbin.org/get"
},
{
"__name__": "k6_http_req_failed_rate",
"expected_response": "true",
"group": "::my group::html",
"method": "GET",
"name": "http://test.k6.io/",
"proto": "HTTP/1.1",
"scenario": "default",
"status": "308",
"url": "http://test.k6.io/"
},
{
"__name__": "k6_http_req_failed_rate",
"expected_response": "true",
"group": "::my group::html",
"method": "GET",
"name": "https://test.k6.io/",
"proto": "HTTP/1.1",
"scenario": "default",
"status": "200",
"tls_version": "tls1.3",
"url": "https://test.k6.io/"
},
{
"__name__": "k6_http_req_failed_rate",
"expected_response": "true",
"group": "::my group::json",
"method": "GET",
"name": "https://httpbin.org/get",
"proto": "HTTP/2.0",
"scenario": "default",
"status": "200",
"tls_version": "tls1.2",
"url": "https://httpbin.org/get"
},
{
"__name__": "k6_http_req_receiving_p99",
"expected_response": "true",
"group": "::my group::html",
"method": "GET",
"name": "http://test.k6.io/",
"proto": "HTTP/1.1",
"scenario": "default",
"status": "308",
"url": "http://test.k6.io/"
},
{
"__name__": "k6_http_req_receiving_p99",
"expected_response": "true",
"group": "::my group::html",
"method": "GET",
"name": "https://test.k6.io/",
"proto": "HTTP/1.1",
"scenario": "default",
"status": "200",
"tls_version": "tls1.3",
"url": "https://test.k6.io/"
},
{
"__name__": "k6_http_req_receiving_p99",
"expected_response": "true",
"group": "::my group::json",
"method": "GET",
"name": "https://httpbin.org/get",
"proto": "HTTP/2.0",
"scenario": "default",
"status": "200",
"tls_version": "tls1.2",
"url": "https://httpbin.org/get"
},
{
"__name__": "k6_http_req_sending_p99",
"expected_response": "true",
"group": "::my group::html",
"method": "GET",
"name": "http://test.k6.io/",
"proto": "HTTP/1.1",
"scenario": "default",
"status": "308",
"url": "http://test.k6.io/"
},
{
"__name__": "k6_http_req_sending_p99",
"expected_response": "true",
"group": "::my group::html",
"method": "GET",
"name": "https://test.k6.io/",
"proto": "HTTP/1.1",
"scenario": "default",
"status": "200",
"tls_version": "tls1.3",
"url": "https://test.k6.io/"
},
{
"__name__": "k6_http_req_sending_p99",
"expected_response": "true",
"group": "::my group::json",
"method": "GET",
"name": "https://httpbin.org/get",
"proto": "HTTP/2.0",
"scenario": "default",
"status": "200",
"tls_version": "tls1.2",
"url": "https://httpbin.org/get"
},
{
"__name__": "k6_http_req_tls_handshaking_p99",
"expected_response": "true",
"group": "::my group::html",
"method": "GET",
"name": "http://test.k6.io/",
"proto": "HTTP/1.1",
"scenario": "default",
"status": "308",
"url": "http://test.k6.io/"
},
{
"__name__": "k6_http_req_tls_handshaking_p99",
"expected_response": "true",
"group": "::my group::html",
"method": "GET",
"name": "https://test.k6.io/",
"proto": "HTTP/1.1",
"scenario": "default",
"status": "200",
"tls_version": "tls1.3",
"url": "https://test.k6.io/"
},
{
"__name__": "k6_http_req_tls_handshaking_p99",
"expected_response": "true",
"group": "::my group::json",
"method": "GET",
"name": "https://httpbin.org/get",
"proto": "HTTP/2.0",
"scenario": "default",
"status": "200",
"tls_version": "tls1.2",
"url": "https://httpbin.org/get"
},
{
"__name__": "k6_http_req_waiting_p99",
"expected_response": "true",
"group": "::my group::html",
"method": "GET",
"name": "http://test.k6.io/",
"proto": "HTTP/1.1",
"scenario": "default",
"status": "308",
"url": "http://test.k6.io/"
},
{
"__name__": "k6_http_req_waiting_p99",
"expected_response": "true",
"group": "::my group::html",
"method": "GET",
"name": "https://test.k6.io/",
"proto": "HTTP/1.1",
"scenario": "default",
"status": "200",
"tls_version": "tls1.3",
"url": "https://test.k6.io/"
},
{
"__name__": "k6_http_req_waiting_p99",
"expected_response": "true",
"group": "::my group::json",
"method": "GET",
"name": "https://httpbin.org/get",
"proto": "HTTP/2.0",
"scenario": "default",
"status": "200",
"tls_version": "tls1.2",
"url": "https://httpbin.org/get"
},
{
"__name__": "k6_http_reqs_total",
"expected_response": "true",
"group": "::my group::html",
"method": "GET",
"name": "http://test.k6.io/",
"proto": "HTTP/1.1",
"scenario": "default",
"status": "308",
"url": "http://test.k6.io/"
},
{
"__name__": "k6_http_reqs_total",
"expected_response": "true",
"group": "::my group::html",
"method": "GET",
"name": "https://test.k6.io/",
"proto": "HTTP/1.1",
"scenario": "default",
"status": "200",
"tls_version": "tls1.3",
"url": "https://test.k6.io/"
},
{
"__name__": "k6_http_reqs_total",
"expected_response": "true",
"group": "::my group::json",
"method": "GET",
"name": "https://httpbin.org/get",
"proto": "HTTP/2.0",
"scenario": "default",
"status": "200",
"tls_version": "tls1.2",
"url": "https://httpbin.org/get"
},
{
"__name__": "k6_iteration_duration_p99",
"scenario": "default"
},
{
"__name__": "k6_iterations_total",
"scenario": "default"
},
{
"__name__": "k6_my_counter_total",
"group": "::my group",
"scenario": "default",
"tag": "test"
},
{
"__name__": "k6_my_rate_rate",
"group": "::my group",
"scenario": "default"
},
{
"__name__": "k6_vus"
},
{
"__name__": "k6_vus_max"
},
{
"__name__": "up",
"instance": "localhost:9090",
"job": "prometheus"
}
]
}
Which is what I then see in my browser.
If the metrics are not there in your Cortex either (we can discard any issues with Grafana querying), did you see any errors in the k6 logs when running the test related to writing the output? Is the test running successfully?
With this information I’m hoping we can keep digging into this.
Cheers!
Hi there @eyeveebe thank you for responding. Apologies for the tardy response. I’m using our corporate cortex appliance not a locally installed one. It appears (thanks to your example) I have a undiagnosable issue with my publicly acquired grafana dashboard JSON 
when I used the dashboard provided in the xk6-output-prometheus-remote/grafana/dashboards/dashboard-results.json file the k6_* key values were retrieved from the dashboard editor query pattern 
so I have no idea how to reproduce the issue I saw with a properly setup dashboard json. json I did not write of course 
the dashboard I used to observe the lack of key values is likely from an ancient blog post - Combine k6 OSS and Prometheus for better observability I can’t say. However my problem appears to be gone.
However, I am having issues with the given dashboard in the that xk6 dashboard. it may not be appropriate for use given that I’m not using the xk6 extension in favor of the 0.42 experimental package. After manually updating the JSON from the xk6 repo dashboard-result.json most of the values display as null. Again after tinkering with the script I get most of the values to appear. Do you know of a dashboard template I can use as a shortcut for learning if not outright copying :).
Hey @PlayStay,
can you explain your current setup in more detail?
From what I understand, you’re running a k6 test on your local machine that’s pushing metrics to a Cortex instance (not plain Prometheus) hosted on your network. And you’re trying to visualize the metrics in a Grafana instance also hosted on your network, by importing the “k6 Test result” dashboard used in the xk6-output-prometheus-remote repository. Correct?
Let me first say that I also confirmed the dashboard works fine in the Docker Compose setup provided by xk6-output-prometheus-remote. I changed the k6 service to use the grafana/k6:0.42.0 image, and Prometheus received all k6 metrics, and the dashboard shows all results:
Given that you’re using a different data source, I think you just need to change the dashboard panels to point to your Cortex data source.
The dashboard uses a DS_PROMETHEUS variable to make this convenient, so you should be able to change it from the default prometheus-in-docker to your Cortex data source:
This is, of course, assuming that the k6 metrics are reaching Cortex, and from your screenshot it seems that some are.
I would start from scratch by recreating the dashboard just in case (without your manual changes to the dashboard JSON), and doing the data source change, and if it doesn’t work, then troubleshoot if metrics are in Cortex to begin with. I’m not familiar with Cortex to help you out there, but you should ask for help outside of a k6 forum. Preferably someone in your team or company would know best about how to configure this properly. From what I understand, Cortex is an API-compatible implementation of Prometheus, so k6 should work transparently with it.
Good luck!
1 Like
yes to all of your initial questions @imiric thanks for responding. also thanks for confirming the use of the dashboard-results template. I have or rather had made the changes to point my source to the cortex instance in my infra but still no go. I kinda figured I’d eventually have to recreate the dashboard from scratch but as you can probably guess I’m being lazy 
.However I’m trying to avoid creating from scratch because I’m Sure I’ll break the cool feature where the dashboard dynamically creates metrics panel automagically from the dashboard variable “testId”.
I’m not sure (yet) why I’m having issues with certain dashboard panels and functions such as irate vs rate not working as intended.
Thanks for the tips I’ll keep chopping until it my screen looks like yours. 
1 Like
Hello @PlayStay! I just wanted to add to the conversation for additional visibility that we’re now publishing the results dashboard in the Grafana community as an “official” dashboard for Prometheus-backed metrics. The actual source for this comes from the dashboard-results.json in the extension repository, which you’ve already been using.
Hopefully, with broader adoption, this dashboard will continue to improve as more folks vet the configuration and provide feedback and updates.
1 Like
That’s excellent sir! I’ll have a look. I appreciate your input.
Best,
Hello folks Imported the json for dashboard - 18030 and got the following error. I forgot to mention explicitly that I’m using the experimental prometheus remote-write k6 version.
k6 run -o experimental-prometheus-rw it does not have the same naming convention for the default metrics as does the xk6 extension exposed metrics in prometheus.
xk6 remote-write dashboard uses _seconds for this metrics
"expr": "histogram_quantile(0.95, sum by(testid) (rate(k6_http_req_duration_seconds{testid=~\"$testid\"}[$__rate_interval])))",
k6 0.42 experimental uses _p99
"expr": "histogram_quantile(0.95, sum by(testid) (rate(k6_http_req_duration_p99{testid=~\"$testid\"}[$__rate_interval])))",
I’ll make the adjustments to the query’s and try try again :). thx.
Hi @PlayStay. Apologies for the late reply! (been traveling and haven’t been checking the forum)
Anyway, I just wanted to mention that there should be no difference between xk6-output-prometheus-remote and the k6 v0.42.0 experimental. The k6 core pulls in the source from the extension directly when it is built.
The metrics renaming had been a more recent change which looks like we need to get updated in the dashboard source. If you’re making the changes for your use, this would be a great opportunity to contribute the fixes! Otherwise, we’ll get them updated and thank you for bringing this issue to our attention!
1 Like
Oooh I’m not worthy to add changes 
. However thanks for responding and I’ll quadruple check to make sure that I’m seeing what I’m seeing.
Cheers @javaducky
1 Like
No worries. A fix is in pending at Correct metric names in dashboard for rates by javaducky · Pull Request #105 · grafana/xk6-output-prometheus-remote · GitHub.
2 Likes
Hi @PlayStay,
the dashboard requires the native histogram feature enabled on the Prometheus server and for k6. You can read how to do it in the specific guide for the metric conversion.
*_p99 are the default value metrics generated for k6 trend metrics without using the native histogram feature. In the future, when the Native Historam feature will be declared stable from the Prometheus project then we may migrate to it as the default logic.
You should get a working dashboard just enabling the native histogram feature.
Let me know if it helps.
2 Likes
Oops. Sorry I didn’t see this document.
Thanks for the tips @codebien.
I’ll have to work with our infra guys to make a change of this type but I’ll reply to this thread when I get some traction.
PlayStay
1 Like
Bad ass!!
Thanks again.
Hi @codebien
I enabled the native histogram feature as below:
However the metrics reporting to Prometheus still ended with *_p99. Any idea? I am on v2.42.
cat prometheus.args
--config.file /usr/local/etc/prometheus.yml
--web.listen-address=127.0.0.1:9090
--storage.tsdb.path /usr/local/var/prometheus
--web.enable-remote-write-receiver
--enable-feature=native-histograms
@codebien Sorry I missed the metric conversion doc that you shared. Looks like I have to enable native histograms on Prometheus side and include the K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM=true env variable both to make it work.
2 Likes