Hi everyone,
We’re running distributed k6 tests using the k6 Operator on Amazon EKS with Karpenter. During the first deployment (cold start), after the initializer pod completes, the operator creates 30 runner pods. While the cluster is provisioning capacity and the runner pods are initializing, the runner count drops from 30 to 26 before the starter pod launches.
The missing runner pods are deleted from the cluster—they don’t remain in a Pending, Failed, or Evicted state.
If we rerun the exact same test a few minutes later, all 30 runner pods start successfully, and the test runs without any issues. We’ve also observed this behavior on On-Demand nodes, so it doesn’t appear to be specific to Spot instances.
My questions are:
-
Is this expected behavior during the first deployment or cold cluster startup?
-
Does the k6 Operator have any reconciliation logic that could delete runner pods if they don’t become ready quickly enough?
-
Has anyone experienced runner pods being deleted during node provisioning or cluster initialization?
-
Are there any recommended configurations or best practices to prevent this behavior?
Any insights or suggestions would be greatly appreciated. TIA.