The hundred users should log in simultaneously only once. How should we declare this in the options object? Any explanations would be helpful

[syedrakeentest11]

export const options = {
vus: 100,
iterations: 100,
};

Is this correct, or should I update it? Can anyone help me? I don’t have any duration condition; I just need 100 users to log in simultaneously, and each user should log in only once.

Please take a look at the metrics shown in the image. I would like to know which metrics are important for evaluating the login results, specifically the minimum, maximum, and average values. Is this approach correct?

Additionally, for further scenarios, I need a token. I am creating a file for each scenario. How can I store the token for future use? Any help to clarify this confusion would be greatly appreciated! Thank you.

[joanlopez]

Hi @syedrakeentest11,

Welcome to our community forums!

Is this correct, or should I update it? Can anyone help me?

Although I cannot completely see your script (perhaps for privacy/confidentiality reasons, I guess), I think your approach is correct, having 100 VUs and 100 iterations should technically perform 100 iterations concurrently, simulating that 100 concurrent users are doing what’s defined on your function.

If you want to make sure that every iteration is using different user credentials, you’ll need to have an array with them (at least 100), and you can pick each one by using the exec.scenario.iterationInTest, as described here and here.

The only thing I’d like to suggest is to revisit the use of SharedArray, so you avoid allocation one copy in memory of the entire JSON file for each VU, but just have a single one shared among all of them. If this is a small/short test, and/or the file is small, it won’t be a problem, but just in case cause in other scenarios the current approach might consume a huge amount of memory.

Please take a look at the metrics shown in the image. I would like to know which metrics are important for evaluating the login results, specifically the minimum, maximum, and average values. Is this approach correct?

That said, let’s jump into the metrics discussion. Although, I’d say that it really depends on the SUT (system under test), and how every iteration looks like (how many steps are involved).

For instance, if the entire function is performing the workflow to log in (despite of the amount of HTTP requests required to do so), and that’s what you want to evaluate, then iteration duration is likely what you want to pay attention to. If no, you way need to tag your requests and/or look at the metrics of the SUT to understand how the different endpoints involved in the operation are being impacted and which one (if any) represents a bottlekneck.

Regarding min, max, avg, etc I’d suggest to revisit what does terms mean statistically (especially percentiles), and to look for some docs around that, cause I don’t think this is the correct place to discuss that, but generally speaking, the near-end percentiles (p90, p95 or even p99) are a good starting point to have a look at, cause they represent the behavior experienced by the huge mass of users, in this case.

Additionally, for further scenarios, I need a token. I am creating a file for each scenario. How can I store the token for future use?

As far as I know, currently there’s no standard (built-in) solution for this, unless you really want to perform all the operations involved as part of each iteration.

What I’d suggest is to consider an external storage, like Redis (GitHub - grafana/xk6-redis: A k6 extension to test the performance of a Redis instance.). I know that eventually, GCk6 will have a solution transparent to users, but it’s not there yet.

Any help to clarify this confusion would be greatly appreciated!

I hope all those explanations help a bit If you feel like you still have some doubts around, just tell me and I’ll try to resolve them

Regards!

[syedrakeentest11]

Hi @joanlopez,

Thank you for the warm welcome!

I’d like to apologize in advance as I cannot share the complete code due to privacy and confidentiality concerns. However, I’m happy to share relevant snippets and clarify my doubts, which I hope will be helpful.

Requirement:
The task is to perform load testing with 50 virtual users (VUs) and ensure the following thresholds:

• P95 response time: Less than 400ms for each API
• Error rate: Less than 10%

The only details provided in the requirement are the above.

Please note that this is a testing environment and not a production environment.

I’m having some doubts, and I’d appreciate your insights.

---

1. Confusion between constant per VU iterations and constant VUs over a duration

Option 1: Constant per VU Iteration

export const options = {
  cloud: {
    distribution: {
      distributionLabel1: { loadZone: 'amazon:in:mumbai', percent: 100 },
    },
  },
  scenarios: {
    homepage_test: {
      executor: 'per-vu-iterations',
      vus: 50,
      iterations: 1,
    },
  },
  thresholds: {
    http_req_duration: ['p(95)<400'],
    http_req_failed: ['rate<0.10'],
  },
};

With this approach, I received a P95 response time of 114ms.

---

Option 2: Constant VUs over a duration

export const options = {
  cloud: {
    distribution: {
      distributionLabel1: { loadZone: 'amazon:in:mumbai', percent: 100 },
    },
  },
  vus: 50, // Number of virtual users
  duration: '1m', // Test duration
  thresholds: {
    http_req_duration: ['p(95)<400'],
    http_req_failed: ['rate<0.10'],
  },
};

With this approach, I received a P95 response time of 811ms.

Question:
Which approach should I use for load testing given the requirement?
For the constant VUs over duration approach, what should I set as the duration since only the number of VUs (50) and thresholds were specified?

---

2. Using JSON for dynamic tokens

I’m using a JSON file containing 100 different account tokens for authorization. Each VU dynamically retrieves and uses a unique token during the test. The JSON file size is around 8 KB because of the dynamic data.

Question:

• Does the size of the JSON or using dynamic tokens impact the metrics significantly?
• Should I instead use the same token for all VUs to simplify the setup, or is using unique tokens better for accurate results?

---

3. Different P95 responses

I’ve noticed that the P95 response time is different for the two approaches:

• Constant per VU iterations: 114ms
• Constant VUs over duration: 811ms

Question:

• How should I interpret these differences?
• Which method is better for the given requirement, and what should I conclude about the performance?

---

If any part of my query is unclear, please let me know, and I’ll be happy to provide more details or share additional code snippets if possible.

Thank you so much for your guidance and support!

Best regards,
Syed Rakeen.

[joanlopez]

Hey @syedrakeentest11,

I’d like to apologize in advance as I cannot share the complete code due to privacy and confidentiality concerns. However, I’m happy to share relevant snippets and clarify my doubts, which I hope will be helpful.

Sure, I’m always happy to help, or try to hehe. Don’t worry about privacy and confidentiality concerns, that’s completely understandable and common.

So, let me try to reply you to each topic discussed on this thread:

1. Confusion between constant per VU iterations and constant VUs over a duration

Here, the difference between both options is that:

• Option 1: Performs 50 operations (your function), distributed across 50 virtual users.
• Option 2: 50 virtual users are performing operations (running your function) during one minute.

So, if the operation would take exactly one minute, then both would be equivalent, but if not, which I think it’s the case, in the second one you’re doing way more than 50 operations.

By looking at your requirements, I think what you’re looking for is Option 1, because that’s what “simulates” the scenario where 50 virtual users perform a operation (e.g. log into a web site).

However, you need to be aware that this is not equivalent to having 50 RPS, because in a realistic environment, when you have 50 users logging into, there’s network delays, human input, etc, etc, and if the operation just takes some ms or a couple of seconds, then there’s no point in time where all the 50 users are doing the operation at exactly the same second.

If you want to achieve a certain rate of RPS, you’ll likely need to explore a different executor, like constant-arrival-rate. But again, I think that looking at your requirements, the Option 1 is likely enough, as it is what’s likely to represent 50 virtual users doing the operation/steps you have in your test function, more realistically.

2. Using JSON for dynamic tokens

• Does the size of the JSON or using dynamic tokens impact the metrics significantly?
• Should I instead use the same token for all VUs to simplify the setup, or is using unique tokens better for accurate results?

Here, I assume you’re only using the JSON “locally”, and that each VU just picks one of the values to send it over HTTP (or whatever transport is used for auth). If that’s the case, I cannot see how that could impact the metrics, it should be fine. Unless having dynamic tokens have any implication for your backend, but not from the k6’s perspective.

Regarding the JSON file size, I’d recommend you to use SharedArray if you’re not doing so yet. But that’s only to keep k6 memory allocations low, nothing that really impacts the metrics of your test.

In any case, 8KBs over 50 VUs shouldn’t be a problem at all. The problem is when you have a larger file, let’s say 5MB, plus a larger amount of VUs, let’s say 10K, because in that case, if you don’t manage memory efficiently, k6 could be allocating those 5MBs up to 10K times, which would be ~50GBs, unlikely to fit a normal laptop memory.

In conclusion, always try to use SharedArray to avoid issues, but in your case, with the rough numbers you shared, you don’t need to care much.

3. Different P95 responses

• How should I interpret these differences?
• Which method is better for the given requirement, and what should I conclude about the performance?

I think I replied to this one above already, but let me try to explain it with further details, and see if that helps you a bit more.

Let’s suppose that what your main test function (export default function { ...}) is to simulate that a user goes to a site, like: mycompany.com/login, types user and password, and clicks on login.

• The Option 1, would be the equivalent of having 50 users doing that operation at a start signal. So, even if they’re all doing the same operation at the same time, the peak of POST /login HTTP requests won’t be 50, because there are different factors that would distribute the load a bit. For instance:
  • Some will take a few seconds to type the user and password, while for some others which are slower typing will take a few more seconds, so not all the 50 users will hit the login button at exactly the same second.
  • Some will have a worse network, so it will take a couple of seconds more for them to load the login page, and so they won’t hit the login button at exactly the same second as those users who have a faster network.
• The Option 2, would be equivalent of having 50 users doing that operation, since a start signal, and during a minute. So, once a user have logged in, let’s say in 5 seconds, they would start the process again: navigate to login page, type the credentials and click on the login button. Repeatedly, over a minute.
  • That’s why here, the peak of RPS in this case is much higher, because the delay caused by the aforementioned reasons would make some users to be performing the operation at the same second, although maybe those that are faster will be on their 5th iteration, while for others it could be their 3rd iteration.

Regarding how to interpret results, and what approach is better:

• The difference of p95 response times clearly indicates that your system is capable to handle the load of 50 virtual users doing an operation one time, but not (well, yes but poorly) when 50 virtual users are doing the same operation over and over for a minute. Otherwise, if the system would be able to handle both scenarios successfully, the p95 response time would be almost the same in both cases, despite of the load.
• If I’d be expecting up to 50 users logging in the same date at the same time (let’s say I launch a promo campaign in my online store), I think I’d just go with Option 1. But it really depends on the use case and what are your expectations.
• Note that “business requirements”, like the aforementioned example with 50 users logging into my online store, are “different” from technical requirements, like, let’s say someone asks to: support 50 clients consuming an HTTP API in parallel (50 RPS). In such case, I’d suggest another execution, as I suggested above.

If any part of my query is unclear, please let me know, and I’ll be happy to provide more details or share additional code snippets if possible.

I think everything is more or less clear. Although, as you can see, I need to guess estimate what your test function is doing, and what kind of requirements do you have. I tried to mention multiple examples, so it helps you idependently of your concrete scenario.

I hope that helps, but if not, just feel free to bring back your remaining doubts again!

[syedrakeentest11]

Hi @joanlopez,

Thank you so much for your detailed and insightful response! It helped me gain a better understanding of the requirements and the distinctions between the two approaches. However, I still have a few additional doubts that I hope you can help clarify.

---

1. Using sleep or random methods during load testing

I now understand that the focus of our requirement is to conduct sustained load testing, so I’ve decided to follow the second option. Below is the snippet I’m currently using:

export const options = {
  cloud: {
    distribution: {
      distributionLabel1: { loadZone: 'amazon:in:mumbai', percent: 100 },
    },
  },
  vus: 50,
  duration: '1m',
  thresholds: {
    http_req_duration: ['p(95)<400'],
    http_req_failed: ['rate<0.10'],
  },
};

My question is:
Should I include sleep (e.g., sleep(1)) or use a random method to introduce delays after each action in the script during load testing? Or would it be better to avoid any sleep methods entirely, considering the focus is on sustained load and observing the system under peak pressure?

---

2. Handling multiple APIs: One file vs. multiple files

In our application, the homepage interacts with multiple APIs. I’m wondering which approach is more suitable for conducting load testing:

Approach 1: One file per API

• For each API, I create a separate test script file. For instance, if there are 5 APIs on the homepage, I create a folder named home under the scripts folder and manually create 5 separate test files, one for each API.
• Each script is run individually, and the threshold is applied per API. For example:

  thresholds: {
    http_req_duration: ['p(95)<400'],
    http_req_failed: ['rate<0.10'],
  }
  

• This method allows me to get the P95 response time for each API separately.

Question:
If I follow this approach, how can I calculate the overall P95 response time across all APIs? Is there a specific calculation I should perform, or should I handle this calculation externally (e.g., aggregating results from each API manually)?

Approach 2: All APIs in one file

• In this method, I write test scripts for all 5 APIs in a single file and run it as one test.
• The thresholds would still apply individually for each API, not globally.

Question:
If I use this method, will I still get the overall P95 response time and individual P95 response times for each API?
Which approach would you recommend for load testing multiple APIs—separate files for each API or combining all APIs into one file?

---

3. Using test scenarios for load testing

Currently, I am not using scenarios in my scripts.

Question:
Is it correct to conduct load testing without defining test scenarios? Should I use scenarios like shared VU iterations to ensure better coverage or more realistic simulation of user behavior?

---

Thank you for taking the time to patiently read through my queries. I truly appreciate your support and guidance in this process. If any part of my questions is unclear, please let me know, and I’ll be happy to provide further details.

Lastly, if there’s a way we can communicate directly (such as via email) for better collaboration, please feel free to share your contact details. Otherwise, no problem at all—I’m happy to continue here.

Thanks once again for your help!

Best regards,
Syed Rakeen.