Hello @chrisellison,
Welcome to our community forums! 
First of all, I’d like to know a bit better about how are you running your test, and how it fails.
My issue is that if our generated file has more than 15k records (json objects) the test tries to initialize and just fails part way through.
When you say it fails, does it mean it fails due to out-of-memory (OOM) issues? Are you running it locally, or in GCk6 (Cloud)?
That said, let’s dive into your specific questions:
- Can K6 handle reading external files of potentially >1GB in size or multiple files when combined are > 1GB? assuming sharedArray is not the correct route
In general, k6 is not very good at handling large files, and it is something we’ve been working on from time to time, and will likely continue to do in the near future. If you’re running your test locally, it will consume some considerable amount of memory, but it should be feasible.
Why do you say SharedArray is not the correct route? Instead, I’d suggest to keep using it but with a few minor changes.
- what is the alternative that best fits this scenario? And how/where might i implement this?
The very first two recommendations I’d suggest are:
- Try to move the data initialization code inside the
SharedArrayinitialization function, as in this example. I see in your example you have the initialization code outside the initialization function, and in that one you just return the global constant defined above (allData).- Note that, as stated in docs: SharedArray is an array-like object that shares the underlying memory between VUs. The function executes only once, and its result is saved in memory once. So, in your example,
allDataisn’t shared but duplicated, and that makes the use ofSharedArraylikely useless. Try to follow the examples.
- Note that, as stated in docs: SharedArray is an array-like object that shares the underlying memory between VUs. The function executes only once, and its result is saved in memory once. So, in your example,
- Try to use the
openfunction from the experimentalfspackage - see these docs. To do so, you mostly need to add one additional import:import { open } from 'k6/experimental/fs';- seeopen.
- if i had skeleton payloads, would it be better to use k6 to update them then use subtle crypto for the message signing and do it in real time e.g. generate + sign the payloads, rather than reading from a file?
Note that large data shared across VUs is read-only, so unless that’s something you would do as part of each iteration, I think your current approach is better. Whether the crypto work should be part of each iteration or not really depends on your use case and scenario, so I cannot tell.
If that would prevent you from having to load large data files, then it might worth a try, because as I said above, k6 is not very good at handling large files.
I hope that helps!