Unexpected end of Stream for multipart request

Grafana k6 OSS Support
1

I am trying to emulate an HTTP POST that occurs when our app uploads a file.

I initially tried a simple http.post as per Multipart request (uploading a file) but that didn’t work (the file was not being uploaded).

Then I tried the FormData polyfill as per the “Advanced multipart request” section.

When that did not work (the file was not being uploaded), I had a closer look at the polyfill and realised that it was not generating the body in the same way as the browser, so I hacked it… a bit -

FormData.prototype.body = function () { let a = []; for (let i = 0; i < this.parts.length; i++) { a.push(this.boundary); let p = this.parts[i]; if (p.file.filename) { a.push('Content-Disposition: form-data; name="' + p.field + '"; filename="' + p.file.filename.replace(/"/g, '%22') + '"'); a.push('Content-Type: ' + (p.file.content_type || 'application/octet-stream')); a.push(''); } else { a.push('Content-Disposition: form-data; name="' + p.field + '"') } a.push(''); a.push(p.file.data); } a.push(${this.boundary}–); let body = []; a.forEach(e => { let data; if(Array.isArray(e)){ console.log('<binary data>'); data = e; }else{ console.log(e); data = this.__toByteArray(e); } body.push(data); body.push(this.__toByteArray('\r\n')); }); return new Uint8Array(body).buffer; };

This also allows me to log the output, which looks pretty much as I need it.

Now I am getting System.IO.IOException: Unexpected end of Stream on the server.

So I looked at it in Wireshark, comparing a request from the browser with a request from k6 (uploading the same file). What I see at the end is -

image
image1471×254 33.7 KB

Also, the browser has File Data: 81740 bytes whereas k6 has File Data: 81739 bytes.

Am I missing something?

3

Are there any k6 devs that can look at this here?

4

Hi @martinp,
can you report the difference you saw between the polyfill (not hacked) and the browser and how you’re calling the FormData code from the script, please? I will look into it.

Thanks.

6

Hi @codebien, apologies for the late reply. I have figured it out and gotten it working now.

After looking at requests in Wireshark a lot, I found that our app does these multipart uploads entirely differently from how the existing k6 form data polyfill (k6fdp) works.

  • our app is sending parts that contain numbers and nulls, k6fdp does not handle these types at all
  • for any part that is not a file, k6fdp appends Content-Type: application/octet-stream, or Content-Type: text/plain for strings, our app does not provide a content-type for anything but the file. Instead, it just converts numbers to strings, null to ‘null’ and provides an array of ASCII character codes for string (which everything but files get converted to).

Here is a fiddle for the formDataPolyfill.js that I used to eventually get it working if you are interested - jsfiddle

7

Hey @martinp,

Instead, it just converts numbers to strings, null to ‘null’ and provides an array of ASCII character codes for string (which everything but files get converted to).

const fd = new FormData();
fd.append('someNumberField', String(2343));
fd.append('someNullField', String(null));

I see these lines working as expected. Do you expect something different?

8

@codebien, here is a sample of what our POSTs look like -

MIME Multipart Media Encapsulation, Type: multipart/form-data, Boundary: "----WebKitFormBoundary0epDUlBBmmVFWAEd"
    [Type: multipart/form-data]
    First boundary: ------WebKitFormBoundary0epDUlBBmmVFWAEd\r\n
    Encapsulated multipart part: 
        Content-Disposition: form-data; name="file-category-code"\r\n\r\n
        Data (15 bytes)
            Data: 636f6e74726163742d75706c6f6164
            [Length: 15]
    Boundary: \r\n------WebKitFormBoundary0epDUlBBmmVFWAEd\r\n
    Encapsulated multipart part: 
        Content-Disposition: form-data; name="contract-id"\r\n\r\n
        Data (1 byte)
            Data: 30
            [Length: 1]
    Boundary: \r\n------WebKitFormBoundary0epDUlBBmmVFWAEd\r\n

I guess that the biggest problem was the assumption that every part would have a declared Content-Type.