K6 Kafka extension - question about reader/writer instantiation in a test with multiple VUs

Grafana k6 Extensions
1

Hi,
I am using the GitHub - mostafa/xk6-kafka: k6 extension to load test Apache Kafka with support for various serialization formats, SASL, TLS, compression, Schema Registry client and beyond and had some trouble understanding how the kafka readers /writers work / get instantiated under the hood.

The use case is - I have to create a reader with different ReaderConfig per VU (in short 1 reader per VU)
In the setup lifecycle I prepare my data for the ReaderConfig and store it in a data structure

{
1 :{
//readerConfig 1
},
2 :{
//readerConfig 2
}

}

The init stage runs once per VU so I define the placeholder for my reader

let kafkaReader = {};

then in the VU stage I instantiate the reader for a VU only for the first iteration

//opts being passed as a parameter to the function in VU stage for the data returned in setup lifecycle

if(exec.vu.iterationInScenario == 0){
    kafkaReader  = new Reader(opts.consumerSettings[exec.vu.idInTest]);
  }
 let messages = kafkaReader.consume({});

The documentation on the git repo- xk6-kafka then suggests that in the teardown lifecycle we should close the reader

export function teardown(data) {
 //close the reader here
}

but teardown just runs once per the whole test. In this case, which reader it is going to close ? I have trouble understanding how the readers work.

Even in the examples on the repo, the reader is instantiated in the init context and init runs once per VU. so my understanding was 1 reader instantiated per VU, so if we allot 2 VUs for a test then there will be 2 readers instantiated.
how would then they get identified in the tear down function and subsequently closed ?

3

Hey @vidishaparab,

Under the hood the kafka-go library is used to manage the connection, batching and all the other stuff. Generally there will many consumers and producers in your system, hence we’re trying to mimic the same behavior when instantiating the objects in the init context. As for the teardown, you’re correct and it will be executed only once per the entire test, but based on experience this will close all the readers, hence committing the offsets and such. Another thing to note is that the Reader is created for (smoke) testing purposes, not load-testing, because you’re probably trying to test your own system that has many readers(, consumers or subscribers), not k6 or the xk6-kafka extension.

If you are still concerned about the resource clean up in your system, try creating and terminating the reader(s) in your VU code. This is a common pattern I’ve seen in user script. However, this might quickly become a bottleneck, as there will be as many connections as there are VUs created. Hence, your Kafka cluster might choke under load. So, use it carefully.

5

Thanks Mostafa. Yes I agree, we are indeed trying to test the subscribers in our system to measure the performance in terms of latency among other things. The reason we wanted to know about reader’s memory utilization because we are trying to run large tests and wanted to identify resource intensive operations to see if there was a way we could optimize the script.

We are currently creating the readers in the VU code, and do it for the first iteration of a VU
t ex. if (exec.vu.iterationInScenario == 0) and since we are using a ramping-arrival-rate executor we would not really know when the last iteration would be so at the moment there is no clean up code for the readers (reader.close()) . Also like you said, creating and terminating the readers every iteration in the VU stage would be a bottleneck as creating the readers come with its own overhead and soon will back fire for a large number of iterations.

Thanks for your prompt reply ! :))

1 Like