I have prometheus remote write enabled, and it seems to be mostly working, but I’m not seeing some http metrics (like http_req_duration).
I enabled native-histograms in Prometheus and enabled K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM in k6.
Any idea why the other metrics are not showing up?
Hi @kenske
Welcome to the community forum 
Can you share a (sanitized) version of your script, and the output you are getting (if you have the end-of-test summary enabled)? I’d like to reproduce this, and it might be related to what the test is doing. Are the http requests finishing?
Thanks!
Thanks @eyeveebe , here’s some more context.
I’m running a sample script I found in the docs:
import http from 'k6/http';
import { check } from 'k6';
export const options = {
stages: [
{ target: 200, duration: '30s' },
{ target: 0, duration: '30s' },
],
};
export default function () {
const result = http.get('https://test-api.k6.io/public/crocodiles/');
check(result, {
'http response status code is 200': result.status === 200,
});
}
I’m using the k6-operator to run distributed tests. These are the logs from one of the pods:
✓ http response status code is 200
checks.........................: 100.00% ✓ 1248 ✗ 0
data_received..................: 1.4 MB 24 kB/s
data_sent......................: 173 kB 2.9 kB/s
http_req_blocked...............: avg=493.11µs min=1.12µs med=2.69µs max=161.36ms p(90)=4.43µs p(95)=5.08ms
http_req_connecting............: avg=208.91µs min=0s med=0s max=100.32ms p(90)=0s p(95)=1.68ms
http_req_duration..............: avg=1.67s min=15.67ms med=1.59s max=3.68s p(90)=3.18s p(95)=3.49s
{ expected_response:true }...: avg=1.67s min=15.67ms med=1.59s max=3.68s p(90)=3.18s p(95)=3.49s
http_req_failed................: 0.00% ✓ 0 ✗ 1248
http_req_receiving.............: avg=90.39µs min=20.49µs med=67.8µs max=21.83ms p(90)=83.45µs p(95)=92.88µs
http_req_sending...............: avg=18.43µs min=6.1µs med=16.13µs max=491.92µs p(90)=22.7µs p(95)=35.83µs
http_req_tls_handshaking.......: avg=260.43µs min=0s med=0s max=60.98ms p(90)=0s p(95)=3.23ms
http_req_waiting...............: avg=1.67s min=15.6ms med=1.59s max=3.68s p(90)=3.18s p(95)=3.49s
http_reqs......................: 1248 20.797054/s
iteration_duration.............: avg=1.67s min=15.79ms med=1.59s max=3.68s p(90)=3.18s p(95)=3.49s
iterations.....................: 1248 20.797054/s
vus............................: 1 min=0 max=67
vus_max........................: 67 min=67 max=67
Stream closed EOF for k6/k6-sample-aigvf-1-sq2g6 (k6)
Here is the k6 resource:
apiVersion: k6.io/v1alpha1
kind: K6
metadata:
name: k6-sample
spec:
parallelism: 3
script:
configMap:
name: k6-tests
file: example.js
arguments: -o experimental-prometheus-rw --tag testid=abc123
runner:
env:
- name: K6_PROMETHEUS_RW_SERVER_URL
value: "http://monitoring-prometheus.monitoring:9090/api/v1/write"
- name: K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM
value: "true"
Hi @kenske !
Sorry for the delay.
I see that you’re using the k6-operator.
Out of curiosity, does just running the k6 standalone (for instance, locally) produce metrics?
K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM=true \
k6 run -o experimental-prometheus-rw --tag testid=3333 script.js
Cheers!
And another question is about the server.
Is it Prometheus? If so, what’s the version, and are native histograms enabled?
E.g. in our docker-compose:
Cheers
I figured out the issue. It’s a problem with my metrics stack. I use Prometheus Operator + Mimir, so I needed to add some extra config settings to Mimir, and Prometheus Operator does not yet support send_native_histograms for remote write (but a fix for this has just been merged).
can use AZURE MANAGED PROMETHEUS AND AZURE MANAGED GRAPHANA FOR THE OUTPUT
Hi @patenvy
Is this a question, or are you pointing out Azure-managed Prometheus and Grafana work? Can you open a separate topic and give more details if it’s a question?
Thanks!
Hello,
I have enabled the native histogram and tried setting the environment variable $env:K6_PROMETHEUS_RW_TREND_STATS = "p(80),p(90),p(95),avg,sum,min,max" as well as $env:K6_PROMETHEUS_RW_TREND_STATS = "count,min,max,p(90),avg", but I am still missing k6_http_req_duration (please check the attached image).
Is there any way to send all the data to Prometheus from k6, similar to generating output with k6 run --out json=test_results.json .\tests\random.test.js?
