Hi all,
I am able to run load test via k6 operator for smaller loads ie rps however when i try to run for larger rps and high number of parallelism i see failures in load test jobs with error
Job has reached the specified backoff limit
Here are the specs I am putting in my k6 operator :
resources:
limits:
cpu: 1000m
memory: 4096Mi
requests:
cpu: 500m
memory: 1024Mi
and my load test scenarios :
export const options = {
scenarios: {
ramping_arrival_rate: {
executor: 'ramping-arrival-rate',
startRate: 20, // Start with 10 iterations/second
timeUnit: '1s', // RPS will be calculated per second
preAllocatedVUs: 10000, // Preallocate 50 VUs
stages: [
{ target: 100, duration: '2m' }, // Ramp up to 500 RPS in 3 minutes
{ target: 1000, duration: '4m' }, // Ramp up to 1000 RPS in the next 2 minutes
{ target: 3000, duration: '3m' }, // Ramp up to 1500 RPS in the next 2 minutes
{ target: 0, duration: '1m' }, // Ramp down to 0 RPS over 2 minutes
],
},
},
};```
i try to run these tests on 10 runners
I get the following response when i try this command
**kubectl describe job load-test-3 -n argo**
Image: 578061096415.dkr.ecr.us-east-1.amazonaws.com/load-test:1.0.0
Port: 6565/TCP
Host Port: 0/TCP
Command:
k6
run
–quiet
–execution-segment=2/10:3/10
–execution-segment-sequence=0,1/10,2/10,3/10,4/10,5/10,6/10,7/10,8/10,9/10,1
–tag
testid=customer-exact-search-2024-10-23
/test/customer-exact-search.js
–address=0.0.0.0:6565
–paused
–tag
instance_id=3
–tag
job_name=load-test-3
Limits:
cpu: 1
memory: 4Gi
Requests:
cpu: 500m
memory: 1Gi
Liveness: http-get http://:6565/v1/status delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:6565/v1/status delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
DB_USERNAME: qqeq
DB_PASSWORD: qeqeq
DB_DATABASE: qqeqeq
DB_HOST: rrr
API_BASE_URL: api.mykaarma.com
TEST_DEALER_UUID: a1dde11c73eca65b529b6125b98c0088dba81e7967f261e5601b0dc64e498ea4
TEST_DEPARTMENT_UUID: ttt
SERVICE_SUSBCRIBER_USERNAME: vhqU7KRiuEy1qfU4TEiYHP-E2zSJgz2CPKEedUPSCwo
SERVICE_SUSBCRIBER_PASSWORD: k1_TunYv4vrgtGCGtc-CFqF99ER2Me8FbVOMwZlIY_M
TEST_DEALER_ID: 1
Mounts:
/test from k6-test-volume (rw)
/tmp from logfolder (rw)
Volumes:
k6-test-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: scripts
Optional: false
logfolder:
Type: HostPath (bare host directory volume)
Path: /var/log/kaarya
HostPathType:
Events:
Type Reason Age From Message
Normal SuccessfulCreate 3m11s job-controller Created pod: load-test-3-5q9h9
Warning BackoffLimitExceeded 3m6s job-controller Job has reached the specified backoff limit
can any one tell me what could be possible reason ? also one tihng i noticed is that i see an extra pod load-test-starter when everything works fine but not when the load-job- pods fails
