Start measure only after warmup

I’m running a staged load test:

  stages: [
    { duration: '5s', target: 50 },
    { duration: '5s', target: 100 },
    { duration: '5s', target: 200 },
    { duration: '5s', target: 400 },
    { duration: '5s', target: 800 },
    { duration: '2m', target: 800 },
  ]

Is it possible to start measuring only after reaching the target of 800 VUs (last stage)?

Hi @noam

Welcome to the community forum :wave:

The easiest way would be to tag the stages. When querying the resulting metrics you can filter by the stage/s that are not in your warm-up.

Can you further elaborate on the reasons behind the behavior you need in the test? I understand you don’t even want k6 to measure the warm-up stages, is that correct? I’m asking in case it’s a feature we should consider, or there is another way to achieve that. We’d like to understand the reasoning behind it. Or tagging the metrics already works for you?

Cheers!

1 Like

Thank you.
Tagging by stage helps - but the end-of-test summary does not include the tags, so they are only useful when looking deeper in the cloud results and/or Prometheus and the like.
Further – when using K6 cloud, the top graph and summary are not affected by tags. A special tag like “excludeFromSummary” whould help here.
Noam.

In case it helps, I had a (kindof) similar requirement, ie discard response times seen during warmup and only count post-warmup stats to determine a pass/fail result.

Not sure if this is the best way, but I did this using two scenarios - a “warmup” and a “loadtest”, based on an approach I saw on a blog post elsewhere.

  scenarios: {
    warmup: {
      executor: 'ramping-vus',
      exec: 'myCustomFunction',
      startVUs: 0,
      stages: [
        { duration: '10', target: LOW_LOAD },
        { duration: '10', target: LOW_LOAD },
      ],
      gracefulStop: '10s',
      startTime: '0s',
    },
    loadTest: {
      executor: 'ramping-vus',
      exec: 'myCustomFunction',
      startVUs: 0,
      stages: [
        { duration: '10s', target: MAX_LOAD },
        { duration: '60s', target: MAX_LOAD },
      ],
      gracefulStop: '5s',
      startTime: '30s',
    },
  },

Both scenarios run essentially the same set of requests (warmup with less load), but the thresholds are set different. For the warmup, I only look at the error rate and abort if it exceeds a preset percentage, and for the loadtest I look at both error rate and response times to determine a pass/fail result, some sample code below:

  thresholds: {
    'myCustomErrorRate{scenario:warmup}': [{ threshold: 'rate<0.1', abortOnFail: true }],
    'myCustomErrorRate{scenario:loadTest}': ['rate<0.05'],
    'myCustomLatency{scenario:loadTest}': ['p(90)<900'],
  },

In the summary report, i remove everything else and only show these, by deleting unneeded keys in the data.metrics in the handleSummary:

→    myCustomErrorRate............: 0.00% ✓ 0        ✗ 236
→    ✓ { scenario:loadTest }........: 0.00% ✓ 0        ✗ 226
→    ✓ { scenario:warmup }..........: 0.00% ✓ 0        ✗ 10
→    myCustomLatency..............: count=236 avg=682.827801 min=466.43075  med=613.865813 max=2455.606668 p(90)=880.20498
→    ✓ { scenario:loadTest }........: count=226 avg=645.225344 min=466.43075  med=609.792042 max=1126.213418 p(90)=800.715188
1 Like

I do this in my API tests. By “failing fast” in the initial scenario, it allows a more intelligent usage of the allotted time within a simplified CI workflow by skipping the full load test as appropriate. I have response time thresholds on both initial and main scenarios but I’m only using abortOnFail for the http_req_failed and checks metrics. Why lose the time for tracking the response times in the main test when it’s already highly likely to fail on outright failing requests. If the problems with meeting thresholds on http_req_failed and checks can be rectified by re-running the pipeline job (i.e. the service/AUT was still in the process of starting all the instances), that’s easy and fast. I am using a slightly longer initial scenario since a few services have many endpoints and the goal is to complete one script iteration. So far I have been using the same request scope but the variety of different data that will be exercised in the initial scenario is highly limited.

1 Like