High Memory Consumption and Low throughput while using Formdata to test file uploads

Grafana k6
1

Hi, we are doing a test involving file upload. The request should be a multipart request and the file size we are using is around 50 MB, although we would like to test with larger files as well. Note that we are running the test using k6 operator in Openshift container platform.

We are referring the docs here - Data Uploads | Grafana k6 documentation for forming the multipart request. When we try the first approach listed in the docs, the test completes successfully but always 20-30% requests fail and the error we get says that the request is not correct (some required property is missing). This is an actual error message -

{“errors”:[{“code”:“connector0012”,“message”:“The provided data for the "File" object isn’t in the correct format for the application. Ensure that the provided data for the operation is valid. Ensure that the properties of the object comply with the definition in the OpenAPI document.”,“error_details”:“Cannot read properties of null (reading ‘name’)”}]}

This suggests that the request was not sent properly and was missing some properties. However, the exact same request is being formed. Please see the code below:

import http from 'k6/http';
import exec from 'k6/execution';
import { uuidv4 } from 'https://jslib.k6.io/k6-utils/1.4.0/index.js';

const fileData = open('./payloads/50mb.txt');

export const options = {
    vus: 2,
    insecureSkipTLSVerify: true,
    duration: "200s"
};


export default function () {

    const url = `xyz.com/file`

    const response = http.post(url, 
        {
            payload: JSON.stringify( {   "name": `multipleStream50mb-${uuidv4()}.mov`,"contentType": "Binary", "fileSize":52428800  }),
            content: http.file(fileData, "50mb.txt", "text/plain"),
        }, 
        {
            headers: {
                'x-xxx-application-credential': 'secret'
            },
        });
    if (response.status > 201){
        console.log('**res**', JSON.stringify(response.body), response.errors, response.status)
    }
}

If we are sending the same request each time, how can it be possible that some properties are missing in some requests? I find this strange.

Then we thought let’s try the Advanced Multipart method using “Formdata” as listed in the docs. So we formed the request as follows -

import http from 'k6/http';
import exec from 'k6/execution';
import { FormData } from 'https://jslib.k6.io/formdata/0.0.2/index.js';
import { uuidv4 } from 'https://jslib.k6.io/k6-utils/1.4.0/index.js';

 export const options = {
  vus: 2,
  insecureSkipTLSVerify: true,
  iterations: 30
};

const file = open('./test/payloads/50mb.txt');



export default async function () {
    const fd = new FormData();

    const payloadJson = JSON.stringify( {   "name": `multipleStream50mb-${uuidv4()}.mov`,"contentType": "Binary", "fileSize":52428800  });

    fd.append('payload', payloadJson);
    fd.append('content', http.file(file, '50mb.txt', 'text/plain'));

    const url = 'xyz.com/file'

    const params = {
        headers: {
        'Content-Type': 'multipart/form-data; boundary=' + fd.boundary,
        'x-xxx-application-credential': 'secret'
    },
  };

    const response = http.post(url, fd.body(), params);
     if (response.status > 201) {
    console.log('**res====**', (typeof(response)), response.status);
  }
}

With this approach, we notice that the memory consumption of the k6 runner pod in kubernetes is extremely high (8-10GB) and the pod crashes with OOM after sometime. To avoid oom, I had to set very high memory limits on my pod.

When I tried this script locally on my mac and checked the memory consumption, it was still very high (8-10GB).

image
image3452×260 76.2 KB

The difference with this advanced multipart approach is that no requests are failing but the throughput drops to extremely low compared with the basic approach. I noticed that k6 is taking a long time to even send the request to backend. It seems that some processing is happening on k6 end.

Here are some other things I have tried -

  1. Tried to open the file as binary by passing the ‘b’ flag in the open function – no difference was seen
  2. Tried to use the experimental fs module but still got the same memory consumption and low throughput.
  3. With a 5mb payload, the advance multipart request seems to be working with no oom crashes but the throughput is still very less.

Is there something wrong that we are doing? Please let me know if any thing is not clear.

I have referred some posts such as the one below while dubugging

Need help - High memory issue