Should I add third party services, css, js images, docs etc in hybrid test

Grafana k6
1

I’m considering a hybrid approach to browser testing and have used the k6 recorder to capture the user’s browser journey.

My hybrid scenario would look like this (separated here for clarity):

export const portfolioProtocolScenario = {
  executor: 'ramping-vus',
  startVUs: 0,
  stages: [
    { duration: '10s', target: 50 }, 
    { duration: '10s', target: 50 },
    { duration: '10s', target: 0 }, 
  ],
  exec: 'portfolioLoad',
  tags: { my_custom_tag: 'protocol' },
};


export const portfolioBrowserScenario = {
  exec: 'e2eBrowserTest', 
  executor: 'per-vu-iterations',
  vus: 5,
  iterations: 5,
  startTime: '15s',
  options: {
    browser: {
      type: 'chromium',
    },
  },
};

As you can see, I’m starting the protocol tests (recorded from the browser) first, and when it reaches peak load, I trigger the browser tests to measure web vitals.

My question is: should I include third-party services, css, js, google analytics, images, docs, etc., captured from the recording in the protocol test?

I’m unsure if it adds value since the browser tests will hit these items again. However, considering the maximum load is on the protocol tests, should I keep them there to ensure nothing goes wrong?

3

Hi @gm4afsal,

Welcome back to the forum! :tada:

That’s a great question, for which I would recommend taking into account multi aspects, as in my humble opinion it’s difficult to give you an answer in general terms.

So, I’d suggest to answer that question considering the kind of third-service we’re speaking about, based on some of the following aspects (complementarily):

  • Availability and dimension; if we’re speaking about a large, world-size, first-class service like Google (e.g. Analytics), then I think it doesn’t make much sense to include those requests in your protocol tests. However, if you’re using a service from a smaller company, or even a self-served one (run by another department within the company), then I’d suggest to include those, at least to some of your tests, to make sure the whole system is capable to handle the load you’re willing to, detecting rate-limits that might need some adjustment, etc.

  • Pricing and associated costs; if we’re speaking about a pay-per-use service, either by subscription or self-hosted (more usage would be translated into higher costs of cpu, memory, etc), then you may want to add those requests in some of your tests (see the aspect above), but definitely not in all of them.

  • Criticality of the service; if we’re speaking about a service that’s strictly required for your system to work, then I’d strongly recommend you to include those at least in some of your tests (or all, depending on the evaluation of the other aspects above). However, if it isn’t strictly needed (e.g. Analytics is usually not critical), then you may add it to some tests, but definitely not to all of them.

Finally, there’s also a few other aspects, probably less important, that you may want to consider. For instance, like how fragility adds to your tests by adding those. For instance, if the requests you make to those third-party services change very often (are dynamic, partly determined at execution time), then it’s probably good idea to include them on browser tests, where they will be naturally executed, but not in protocol tests, where they are probably just a snapshot of an older execution.

So, to sum up, I’d recommend you going over all of your third-party services and carefully analyzing some of those aspects, in order to figure out whether it makes sense to include them into your protocol tests or not. In general, for non-critical, widely available services like Analytics, I think it makes little sense, but on case-by-case basis, it really depends.

I hope that helps! :bowing_man: