Missing K6 Metrics (P95 Response Time) in Grafana Dashboard

Dear All,

I am pushing my load-tests result to Grafana via Prometheus, and the push is successful because I can see the data in Grafana Dashboard.
But there is one Metrics missing or showing 0 result:

I am using below configuration:

      K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM: true
      K6_PROMETHEUS_RW_TREND_STATS: p(80),p(90),p(95),avg,sum,min,max,rate

And

summaryTrendStats: ['avg', 'min', 'med', 'max', 'p(95)', 'p(99)', 'p(99.99)', 'count']

This is the query used inside Grafana:

histogram_quantile(0.95, sum by(testid) (rate(k6_http_req_duration_seconds{testid=~"$testid"}[$__rate_interval])))

Could you please suggest whats wrong i am doing here or any suggestion/solution?

Thanks!

1 Like

Hi @ramandeep2008singh,
are you following the instructions in the documentation?

Are you using the reported env variables at the same time? Did you find it in some documentation? If yes, we may need to update it because it is not expected.

Let me know if it helps.

Hi @codebien
Thank you very much for reply on our query.

Yes, We followed the same documentation.
we are triggering tests through github actions workflow and we set below are the env variables to send test results to prometheus remote write.

K6_PROMETHEUS_RW_USERNAME: ${{ secrets.K6_PROMETHEUS_RW_USERNAME }}
K6_PROMETHEUS_RW_PASSWORD: ${{ secrets.K6_PROMETHEUS_RW_PASSWORD }}
K6_PROMETHEUS_RW_SERVER_URL: ${{ secrets.K6_PROMETHEUS_RW_SERVER_URL }}
K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM: true
K6_PROMETHEUS_RW_TREND_STATS: p(80),p(90),p(95),avg,sum,min,max,count

We are also using official k6 dashboard to monitoring results, this dashboard has several panels with metric k6_http_req_duration_seconds

For example:

histogram_quantile(0.95, sum by(testid) (rate(k6_http_req_duration_seconds{testid=~"$testid"}[$__rate_interval])))

Runner output at the end of test is look like below (taken from one of the tests execution with similar config)


     checks.........................: 100.00% ✓ 1641     ✗ 0   
     data_received..................: 1.1 MB  2.5 kB/s
     data_sent......................: 368 kB  871 B/s
     http_req_blocked...............: avg=362.7µs  min=269ns    med=354ns   max=428.71ms p(90)=576ns   p(95)=670ns   
     http_req_connecting............: avg=9.47µs   min=0s       med=0s      max=1.17ms   p(90)=0s      p(95)=0s      
     http_req_duration..............: avg=4.42s    min=589.76ms med=3.96s   max=11.16s   p(90)=7.77s   p(95)=8.51s   
       { expected_response:true }...: avg=4.42s    min=589.76ms med=3.96s   max=11.16s   p(90)=7.77s   p(95)=8.51s   
     http_req_failed................: 0.00%   ✓ 0        ✗ 1642
     http_req_receiving.............: avg=78.45µs  min=35.65µs  med=69.47µs max=488.77µs p(90)=98.05µs p(95)=135.84µs
     http_req_sending...............: avg=76.26µs  min=36.82µs  med=61.76µs max=1.28ms   p(90)=96.27µs p(95)=144.73µs
     http_req_tls_handshaking.......: avg=347.73µs min=0s       med=0s      max=425.65ms p(90)=0s      p(95)=0s      
     http_req_waiting...............: avg=4.42s    min=589.62ms med=3.96s   max=11.16s   p(90)=7.77s   p(95)=8.51s   
     http_reqs......................: 1642    3.888249/s
     iteration_duration.............: avg=4.42s    min=589.97ms med=3.96s   max=11.16s   p(90)=7.77s   p(95)=8.51s   
     iterations.....................: 1641    3.885881/s
     vus............................: 1       min=0      max=20
     vus_max........................: 20      min=20     max=20


and Available metrics for K6 through grafana cloud after the test is finished are below

k6_data_received_total
k6_data_sent_total
k6_http_req_blocked_p99
k6_http_req_connecting_p99
k6_http_req_duration_p99
k6_http_req_failed_rate
k6_http_req_receiving_p99
k6_http_req_sending_p99
k6_http_req_tls_handshaking_p99
k6_http_req_waiting_p99
k6_http_reqs_total
k6_iteration_duration_p99
k6_iterations_total
k6_vus
k6_vus_max

If we compare the panel query with available metrics in grafana from prometheus data source we observe that either the query is using incorrect metric or we are unable to generate the required metric through our tests. please let us know if we need enable any env variables for generating required metric

Thank you.

Hi @srinivas.sambari,
as said you should not use the at the same time the Native Histogram and the Trend stats conversions. They are exclusive, and as you can see from the reported metrics list, the output is generating only the Trend stats.

The k6 dashboard works only with the Native Histogram so you have to expect a metric like k6_http_req_duration_seconds as the query mentions.

You should try removing the K6_PROMETHEUS_RW_TREND_STATS option.

Let me know if it helps.

Hi @codebien
we have added this env K6_PROMETHEUS_RW_TREND_STATS later when found the metric k6_http_req_duration_seconds is not coming through results.

Also just now we ran a test by removing the K6_PROMETHEUS_RW_TREND_STATS env variable and found Response time is coming as zero only, there is no change, and below is the result from runner after test is completed.

checks.........................: 100.00% ✓ 1798     ✗ 0   
     data_received..................: 1.1 MB  2.7 kB/s
     data_sent......................: 400 kB  947 B/s
     http_req_blocked...............: avg=311.39µs min=283ns    med=357ns   max=431.36ms p(95)=664ns    p(99)=5.15ms   p(99.99)=356.25ms count=1799
     http_req_connecting............: avg=7.46µs   min=0s       med=0s      max=1.09ms   p(95)=0s       p(99)=296.52µs p(99.99)=1.05ms   count=1799
     http_req_duration..............: avg=4.04s    min=413.77ms med=4.09s   max=9.02s    p(95)=7.41s    p(99)=8.2s     p(99.99)=8.98s    count=1799
       { expected_response:true }...: avg=4.04s    min=413.77ms med=4.09s   max=9.02s    p(95)=7.41s    p(99)=8.2s     p(99.99)=8.98s    count=1799
     http_req_failed................: 0.00%   ✓ 0        ✗ 1799
     http_req_receiving.............: avg=81.07µs  min=37.63µs  med=69.61µs max=1.2ms    p(95)=136.04µs p(99)=301.56µs p(99.99)=1.11ms   count=1799
     http_req_sending...............: avg=73.3µs   min=30.87µs  med=61.1µs  max=1.36ms   p(95)=142.2µs  p(99)=266.43µs p(99.99)=1.29ms   count=1799
     http_req_tls_handshaking.......: avg=296.88µs min=0s       med=0s      max=426.08ms p(95)=0s       p(99)=4.32ms   p(99.99)=351.75ms count=1799
     http_req_waiting...............: avg=4.04s    min=413.61ms med=4.09s   max=9.02s    p(95)=7.41s    p(99)=8.2s     p(99.99)=8.98s    count=1799
     http_reqs......................: 1799    4.258285/s
     iteration_duration.............: avg=4.05s    min=413.97ms med=4.09s   max=9.02s    p(95)=7.41s    p(99)=8.2s     p(99.99)=8.98s    count=1799
     iterations.....................: 1798    4.255918/s
     vus............................: 1       min=0      max=20
     vus_max........................: 20      min=20     max=20

Did you enable the experimental Native Histogram feature on the Prometheus server?

Can you also report the k6, Prometheus and Grafana versions you’re using, please?

Hi @codebien
As said earlier we are writing metrics to prometheus remote write in this case it is the prometheus endpoint part of grafana cloud subscription.

our Grafana cloud is running on version Grafana v10.0.2-cloud.1.94a6f396 (b2bbe10fbc)

Hi @srinivas.sambari,
Native Histogram is an experimental feature for Prometheus. Why do you expect it to work?
As far as I know, Grafana Cloud doesn’t support yet this feature. You may try to contact Grafana Cloud support and ask if they can enable the feature for you.

Let me know if it helps.

make sure you have this option on when you run Prometheus:

–enable-feature=native-histograms

Then it should work when you run K6 with env variables:
K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM
and
K6_PROMETHEUS_RW_TREND_STATS

and with this option on k6 run -o experimental-prometheus-rw

We raise a request to check with Grafana Cloud about the native histogram is enabled or not.

Apart from this, we gave a try in local by enabling native histogram config in prometheus instance and then run the tests locally and sending metrics to local prometheus server.
also found this metric is not coming in local.

Here we gave a try by setting

  1. both K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM and K6_PROMETHEUS_RW_TREND_STATS env variables at the same time found results were same
  2. in next iteration we unset the K6_PROMETHEUS_RW_TREND_STATS and set only K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM env variable also get the same results here
sss@ip-192-xx-x-xx Documents % export K6_PROMETHEUS_RW_TREND_STATS=p(80),p(90),p(95),avg,sum,min,max,med,count
sss@ip-192-xx-x-xx Documents % export K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM=true                                                                          
sss@ip-192-xx-x-xx Documents % K6_PROMETHEUS_RW_SERVER_URL=http://localhost:9090/api/v1/write k6 run -o experimental-prometheus-rw onboardingServiceLoadTest.js

          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: onboardingServiceLoadTest.js
     output: Prometheus remote write (http://localhost:9090/api/v1/write)

  scenarios: (100.00%) 1 scenario, 5 max VUs, 5m30s max duration (incl. graceful stop):
           * default: Up to 5 looping VUs for 5m0s over 3 stages (gracefulRampDown: 30s, gracefulStop: 30s)

INFO[0002] Request succeeded!                            source=console
INFO[0002] 200                                           source=console

     ✓ is status 201

     █ setup

     checks.........................: 100.00% ✓ 1107     ✗ 0   
     data_received..................: 665 kB  2.2 kB/s
     data_sent......................: 237 kB  783 B/s
     http_req_blocked...............: avg=794.85µs min=0s       med=1µs     max=533.49ms p(95)=3µs      p(99)=3µs    p(99.99)=484.34ms count=1108
     http_req_connecting............: avg=100.38µs min=0s       med=0s      max=24.12ms  p(95)=0s       p(99)=0s     p(99.99)=23.76ms  count=1108
     http_req_duration..............: avg=1.11s    min=505.76ms med=1.07s   max=2.77s    p(95)=1.75s    p(99)=2.47s  p(99.99)=2.76s    count=1108
       { expected_response:true }...: avg=1.11s    min=505.76ms med=1.07s   max=2.77s    p(95)=1.75s    p(99)=2.47s  p(99.99)=2.76s    count=1108
     http_req_failed................: 0.00%   ✓ 0        ✗ 1108
     http_req_receiving.............: avg=221.83µs min=16µs     med=134.5µs max=35.89ms  p(95)=370.69µs p(99)=1.49ms p(99.99)=32.92ms  count=1108
     http_req_sending...............: avg=247.12µs min=21µs     med=166µs   max=11.17ms  p(95)=513.19µs p(99)=1.73ms p(99.99)=10.74ms  count=1108
     http_req_tls_handshaking.......: avg=590.44µs min=0s       med=0s      max=436.72ms p(95)=0s       p(99)=0s     p(99.99)=394.3ms  count=1108
     http_req_waiting...............: avg=1.11s    min=505.23ms med=1.07s   max=2.77s    p(95)=1.75s    p(99)=2.47s  p(99.99)=2.76s    count=1108
     http_reqs......................: 1108    3.665183/s
     iteration_duration.............: avg=1.11s    min=506.58ms med=1.07s   max=2.77s    p(95)=1.76s    p(99)=2.47s  p(99.99)=2.76s    count=1108
     iterations.....................: 1107    3.661876/s
     vus............................: 1       min=0      max=5 
     vus_max........................: 5       min=5      max=5 


running (5m02.3s), 0/5 VUs, 1107 complete and 0 interrupted iterations
default ✓ [======================================] 0/5 VUs  5m0s
sss@ip-192-xx-x-xx Documents % unset K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM
sss@ip-192-xx-x-xx Documents % clear                                                                                                                           



sss@ip-192-xx-x-xx Documents % export K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM=true                                                                          
sss@ip-192-xx-x-xx Documents % K6_PROMETHEUS_RW_SERVER_URL=http://localhost:9090/api/v1/write k6 run -o experimental-prometheus-rw onboardingServiceLoadTest.js

          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: onboardingServiceLoadTest.js
     output: Prometheus remote write (http://localhost:9090/api/v1/write)

  scenarios: (100.00%) 1 scenario, 5 max VUs, 5m30s max duration (incl. graceful stop):
           * default: Up to 5 looping VUs for 5m0s over 3 stages (gracefulRampDown: 30s, gracefulStop: 30s)

INFO[0001] Request succeeded!                            source=console
INFO[0001] 200                                           source=console

     ✓ is status 201

     █ setup

     checks.........................: 100.00% ✓ 1563     ✗ 0   
     data_received..................: 924 kB  3.1 kB/s
     data_sent......................: 331 kB  1.1 kB/s
     http_req_blocked...............: avg=570.74µs min=0s       med=1µs      max=510.91ms p(95)=3µs      p(99)=3µs    p(99.99)=449.41ms count=1564
     http_req_connecting............: avg=54.93µs  min=0s       med=0s       max=17.57ms  p(95)=0s       p(99)=0s     p(99.99)=17.54ms  count=1564
     http_req_duration..............: avg=788.05ms min=391.38ms med=730.78ms max=2.76s    p(95)=1.24s    p(99)=2.05s  p(99.99)=2.76s    count=1564
       { expected_response:true }...: avg=788.05ms min=391.38ms med=730.78ms max=2.76s    p(95)=1.24s    p(99)=2.05s  p(99.99)=2.76s    count=1564
     http_req_failed................: 0.00%   ✓ 0        ✗ 1564
     http_req_receiving.............: avg=207.7µs  min=23µs     med=138µs    max=17.18ms  p(95)=346.84µs p(99)=1.54ms p(99.99)=16.89ms  count=1564
     http_req_sending...............: avg=266.39µs min=39µs     med=176µs    max=5.31ms   p(95)=625.79µs p(99)=2.1ms  p(99.99)=5.28ms   count=1564
     http_req_tls_handshaking.......: avg=464.03µs min=0s       med=0s       max=452.52ms p(95)=0s       p(99)=0s     p(99.99)=393.7ms  count=1564
     http_req_waiting...............: avg=787.57ms min=391.14ms med=730.4ms  max=2.76s    p(95)=1.24s    p(99)=2.05s  p(99.99)=2.76s    count=1564
     http_reqs......................: 1564    5.182447/s
     iteration_duration.............: avg=789.12ms min=391.77ms med=731.27ms max=2.76s    p(95)=1.25s    p(99)=2.05s  p(99.99)=2.76s    count=1564
     iterations.....................: 1563    5.179133/s
     vus............................: 1       min=0      max=5 
     vus_max........................: 5       min=5      max=5 


running (5m01.8s), 0/5 VUs, 1563 complete and 0 interrupted iterations
default ✓ [======================================] 0/5 VUs  5m0s
sss@ip-192-xx-x-xx Documents % 

I see you did
unset K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM and not TREND_STATS

Please, if you want to use the dashboard don’t set at all the TREND_STATS variable.

Which version are you running locally of Prometheus? Do you see the other metrics? Are you following the guide Prometheus remote write; I don’t see the test id set from your commands.

Hi @codebien

Here is the clear output of CLI

xxi@ip-192-xx-x-xxx Documents % echo $K6_PROMETHEUS_RW_TREND_STATS
p(80),p(90),p(95),avg,sum,min,max,med,count
xxi@ip-192-xx-x-xxx Documents % unset K6_PROMETHEUS_RW_TREND_STATS
xxi@ip-192-xx-x-xxx Documents % echo $K6_PROMETHEUS_RW_TREND_STATS

xxi@ip-192-xx-x-xxx Documents % clear


xxi@ip-192-xx-x-xxx Documents % export K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM=true                                                                          
xxi@ip-192-xx-x-xxx Documents % echo $K6_PROMETHEUS_RW_TREND_STATS 

xxi@ip-192-xx-x-xxx Documents % echo $K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM
true
xxi@ip-192-xx-x-xxx Documents % clear



xxi@ip-192-xx-x-xxx Documents % K6_PROMETHEUS_RW_SERVER_URL=http://localhost:9090/api/v1/write k6 run -o experimental-prometheus-rw onboardingServiceLoadTest.js

          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: onboardingServiceLoadTest.js
     output: Prometheus remote write (http://localhost:9090/api/v1/write)

  scenarios: (100.00%) 1 scenario, 5 max VUs, 5m30s max duration (incl. graceful stop):
           * default: Up to 5 looping VUs for 5m0s over 3 stages (gracefulRampDown: 30s, gracefulStop: 30s)

INFO[0001] Request succeeded!                            source=console
INFO[0001] 200                                           source=console

     ✓ is status 201

     █ setup

     checks.........................: 100.00% ✓ 859      ✗ 0  
     data_received..................: 525 kB  1.7 kB/s
     data_sent......................: 186 kB  614 B/s
     http_req_blocked...............: avg=1.03ms   min=0s       med=1µs   max=471.35ms p(90)=1µs     p(95)=2µs     
     http_req_connecting............: avg=151.72µs min=0s       med=0s    max=44.41ms  p(90)=0s      p(95)=0s      
     http_req_duration..............: avg=1.43s    min=363.17ms med=1.5s  max=2.82s    p(90)=2.05s   p(95)=2.24s   
       { expected_response:true }...: avg=1.43s    min=363.17ms med=1.5s  max=2.82s    p(90)=2.05s   p(95)=2.24s   
     http_req_failed................: 0.00%   ✓ 0        ✗ 860
     http_req_receiving.............: avg=160.51µs min=36µs     med=99µs  max=20.13ms  p(90)=199.1µs p(95)=303.04µs
     http_req_sending...............: avg=165.64µs min=37µs     med=114µs max=7.42ms   p(90)=232.2µs p(95)=320.19µs
     http_req_tls_handshaking.......: avg=785.65µs min=0s       med=0s    max=429.68ms p(90)=0s      p(95)=0s      
     http_req_waiting...............: avg=1.43s    min=362.89ms med=1.5s  max=2.82s    p(90)=2.05s   p(95)=2.24s   
     http_reqs......................: 860     2.845267/s
     iteration_duration.............: avg=1.43s    min=363.42ms med=1.5s  max=2.82s    p(90)=2.05s   p(95)=2.24s   
     iterations.....................: 859     2.841958/s
     vus............................: 1       min=0      max=5
     vus_max........................: 5       min=5      max=5


running (5m02.3s), 0/5 VUs, 859 complete and 0 interrupted iterations
default ✓ [======================================] 0/5 VUs  5m0s

xxi@ip-192-xx-x-xxx Documents % 

Also we have tags given inside the test script like below

export const options = {
    stages: [
        { duration: rampUpDuration, target: maxVirtualUsers }, 
        { duration: durationHold, target: maxVirtualUsers }, 
        { duration: rampDownDuration, target: 0 }, 
    ],
    tags: {
        testid: 'xxx-service',
        scenario: 'load-tests - ' + scenario + '_' + stringDate + '_' + time,
    },

}

we see testId and scenario both are coming fine including other metrics like rps, but RT is missing.

our prometheus deployment through docker compose and here is the docker file used for it which pulls the latest prometheus image.

FROM prom/prometheus
ADD prometheus.yml /etc/prometheus/
ENTRYPOINT [ "/bin/prometheus" ]
CMD        [ "--web.listen-address=0.0.0.0:9090", \
             "--config.file=/etc/prometheus/prometheus.yml", \
             "--storage.tsdb.path=/prometheus", \
             "--web.console.libraries=/usr/share/prometheus/console_libraries", \
             "--web.console.templates=/usr/share/prometheus/consoles", \
             "--web.enable-admin-api", \
             "--web.enable-remote-write-receiver",  \
             "--enable-feature=native-histograms,remote-write-receiver" ]

here is the result from dashboard

Do you open the dashboard from the test list dashboard by selecting the <testid>?

More suggestions:

Check if the metric is in the metric explorer.
Check if you have any errors in the Prometheus logs.

Hi @codebien @dlight
Thank you very much for your support here, We have raised a ticket for Grafana cloud for enabling Native histogram support.

After this feature is enabled for us, we see the Dashboard is filled with all the required fields including ResponseTime Metric.

Once again Thank you very much for all the support.

1 Like