I have a specific testing request to verify the limit for the number of api requests to an endpoint in a min and was wondering if k6 can help me validate it?
Can someone pls suggest?
Thanks!
Hi @21.prash
Here are some thoughts:
- To be honest I’m not sure I fully understand your goal: do you want to prove that the endpoint can exceed a predefined request rate limit or will not exceed it? You may want to review closed vs. open model and executor types to suit the objective.
- Most well-behaved load tests will have some delay or sleep between requests, otherwise you are just hammering the application. That being said, with a decent quantity of virtual users, even when employing sleep it will be hammering anyhow, but the iteration pacing of each thread will be less frantic. Mainly I mention this to consider the effect against your objective.
- The standard results output already contains the request rate metric and if your test has only one endpoint, no additional actions need to be taken to specifically track it. You will receive the http_reqs metric and it is expressed in requests per second.
- For codified/programmatic grading, you will need to define a Threshold against some metric in the results. With a single request in the function you should be able to use the built-in metrics and not require a custom metric to differentiate. Thresholds are defined in test options. Try to set one on the
http_reqsbuilt-in-metric. - If your test has multiple requests (endpoints), you will need to differentiate the target endpoint from any other data in the output, so you will need to define a custom metric of Counter type and reference this custom counter using Counter.add() on your target endpoint. Defining the counter will generate a new separate line to the test output, tracking your request by total numerical count and count by second. (Rate type metric may also work, I have never used it.). Then as before, you define a threshold to grade that metric, but take note that you will need to use your custom metric name in the threshold expression instead of the built-in
http_reqsmetric. - You can configure the test for maximum duration of 1 minute. That could provide a short general smoke test of your logic and perhaps a close approximation of your very specific objective, but starting from zero entails a rapid launch of connections and requests which may skew what is possible in that short time frame. I personally prefer longer tests to more gracefully arrive at the “steady state” rate and then measure. You can employ stages within options with time-constrained durations for ramp-up, maintaining max load, then ramping down.
- If you want to test for highest possible rate and then grade the actual results against a target value, you can use the “ramping-vus” executor type, here is an example with stages. To effect your grading by excluding the rampup stage, you can tag the stage section you want and then have an if condition on the endpoint to only run your your counter if the identified stage is running.
- If you are specifically intending to set a governor that the application can meet but not exceed a rate, set up a scenario to run your test function and use the “constant-arrival-rate” scenario executor instead a VU-based executor. The documentation explains how virtual users get allocated to achieve this, which might require some experimentation. Note that this executor rate configuration uses iterations (loop of the main function) per time unit, not requests. If the function being executed has just the one endpoint, then iteration rate will equal request rate.
Snippet of results output from one of my tests showing set of custom counter metrics against different endpoints.
...
...
Tx_Counter_StandardsWorksApiController_getCommitteeById..............: 431991 4.99575/s
Tx_Counter_StandardsWorksApiController_getCommitteeParticipantById...: 432004 4.9959/s
Tx_Counter_StandardsWorksApiController_getCommitteeParticipants......: 432004 4.9959/s
Tx_Counter_StandardsWorksApiController_getCommitteeResources.........: 431983 4.995657/s
Tx_Counter_StandardsWorksApiController_getCommittees.................: 2159915 24.978286/s
Tx_Counter_StandardsWorksApiController_getRequests...................: 1295912 14.986544/s
Tx_Counter_StandardsWorksApiController_getRoles......................: 432004 4.9959/s
Tx_Counter_StandardsWorksApiController_getSecurity...................: 432004 4.9959/s
...
...
@21.prash Before you go through my overly-long first reply, check out this article which show thresholds on the built-in http_reqs metric and a custom metric of Counter type. The options in these examples show the threshold syntax, but you will still need to have other logic to constrain the test for one minute, determine the model of your load generation, etc.
Hello @richmarshall - I really appreciate your effort to help me with this question!
I understand now that there are two metrics that can track the number of requests
http_reqs and counter.
Lets assume that there are only two users who ramp at from the beginning of the test. And they each fire three http requests.
How can I add a check to guard that they do not cross a specific rate.
@21.prash
I’m sure with some underlying practical limitations, you can define however many custom Counter metrics that you might need to. There are 4 custom types available of which Counter is one.
2 users x 3 http requests = 6 requests. What kind of load test is this? I still do not understand exactly what you are trying to do or need to do, but it sounds like users that are running some asynchronous request or maybe running a long running report. Is the test doing anything else at all besides these?
If you have implemented your assumption as written, you already know how many requests they will fire, or you are somehow able to control it, so why is the check/guard required? Probably easier than writing a loop within the main test function is to use the executor type of “Per VU iterations”, here’s the doc overview: “With the per-vu-iterations executor, each VU executes an exact number of iterations. The total number of completed iterations equals vus * iterations.” So: using this executor does not require special logic to perform the 2x3 check - if you configure the scenario with 2 users by 3 iterations the test will end on its own after the 6 total requests (the 2 user threads will each run their 3 iterations in parallel).
Hello @richmarshall - I finally got answers to most of my questions on the requirement.
So the requirement is that we limit 100 api requests within a min and I need to check if that holds true or not.
I need to know if there is a way I can delay k6 script to start at the beginning of the next min so that i time it to start at 0th sec of the next min. I know there is “startTime” under options but can i programmatically pass a var there? Or is there a better solution?
I can do this without k6 as well but I want to use the builtin mechanism of VUs in my scripts which otherwise can be complicated without k6 (to use multi threading).
My approach is to check the response of every request and maintain a counter and the moment it crosses 100 I expect the script to receive 400.
Hence, the second question is can I programmatically stop the execution once I do not get an expected response?
TIA!
@21.prash Yes, if you have startTime you can offset the start of a scenario after other parts of the test have begun. You can pass a static variable to options fields and even compute them dynamically. In some of my tests, I do not start the actual multi-threaded load test until 60 seconds after start of the overall test. In one scenario beginning immediately the first 55 seconds is for a functional/smoke test as a single thread with 5 seconds for gracefulStop, then assuming that passes all its key thresholds the actual load test starts at 60 seconds.
Assuming there is something different your test is doing in parallel, I think the path of the least amount of code to control your (max) 100 requests/minute is to keep this request in an isolated scenario with no other requests. I would review this table of executor types for their exact options.
Looking at your revised requirement, the per-vu-iterations executor type may be the closest match if you must observe a max 100 requests within a 1 minute time frame. You can configure the number of VUs to accomplish that work. This executor should eliminate the need to write code for the loop of the requests. (This may not be what you want, as AFAIK it will literally stop at 100.)
As each request runs it sounds like you will need to have implemented a counter for the # of pass vs. fail, and you should be able to set up an if condition to stop if 400 occurs. Maybe the simplest way to control the stop is from a threshold (see this example but you will need to add the abortOnFail property to that threshold.
Hello @richmarshall - Apologies for coming on this late. I had other priority tickets to finish before this.
My check is not to have exactly 100 api requests in a min, but to prove that 101th api request within a min fail with 400. So I do not have the problem to time the run. I wrote a shell script that sleeps for the remaining secs in a min before it starts execution so the timing is taken care of.
What I want to know is how to fail the test when the 101th api request is not 404? I saw abortOnFail property under threshold but it needs a threshold metric (or a check) I dont have any need to check the response time or any other check.
The only thing that I am struggling with is how to abort the test when the 101th request is not 400?
See Test Abort
Hello @richmarshall - I got the answer! Thank you so much! I used the counter to count positive responses and user the abortOnFail property like below:
thresholds:
{
'positive_response_counter':
[
{
abortOnFail: true,
threshold: 'count <= 100', // 100 or fewer positive responses
}
],
}
My POC seems working perfectly! TYSM!! 
1 Like