Appending ArrayBuffers

Question

What is the preferable way of appending/combining ArrayBuffers?

I'm receiving and parsing network packets with a variety of data structures. Incoming messages are read into ArrayBuffers. If a partial packet arrives I need to store it and wait for the next message before re-attempting to parse it.

Currently I'm doing something like this:

function appendBuffer( buffer1, buffer2 ) {
  var tmp = new Uint8Array( buffer1.byteLength + buffer2.byteLength );
  tmp.set( new Uint8Array( buffer1 ), 0 );
  tmp.set( new Uint8Array( buffer2 ), buffer1.byteLength );
  return tmp.buffer;
}

Obviously you can't get around having to create a new buffer as ArrayBuffers are of a fixed length, but is it necessary to initialize typed arrays? Upon arrival I just want is to be able to treat the buffers as buffers; types and structures are of no concern.

Answer 1

You could always use DataView (http://www.khronos.org/registry/typedarray/specs/latest/#8) rather than a specific typed array, but as has been mentioned in the comments to your question, you can't actually do much with ArrayBuffer on its own.

Answer 2

It seems you've already concluded that there is no way around creating a new array buffer. However, for performance sake, it could be beneficial to append the contents of the buffer to a standard array object, then create a new array buffer or typed array from that.

var data = [];

function receive_buffer(buffer) {
    var i, len = data.length;

    for(i = 0; i < buffer.length; i++)
        data[len + i] = buffer[i];

    if( buffer_stream_done()) 
        callback( new Uint8Array(data));
}

Most javascript engines will already have some space set aside for dynamically allocated memory. This method will utilize that space instead of creating numerous new memory allocations, which can be a performance killer inside the operating system kernel. On top of that you'll also shave off a few function calls.

A second, more involved option would be to allocate the memory beforehand. If you know the maximum size of any data stream then you could create an array buffer of that size, fill it up (partially if necessary) then empty it when done.

Finally, if performance is your primary goal, and you know the maximum packet size (instead of the entire stream) then start out with a handful of array buffers of that size. As you fill up your pre-allocated memory, create new buffers between network calls -- asynchronously if possible.

Answer 3

function concat (views: ArrayBufferView[]) {
    let length = 0
    for (const v of views)
        length += v.byteLength
        
    let buf = new Uint8Array(length)
    let offset = 0
    for (const v of views) {
        const uint8view = new Uint8Array(v.buffer, v.byteOffset, v.byteLength)
        buf.set(uint8view, offset)
        offset += uint8view.byteLength
    }
    
    return buf
}

Answer 4

Why not using a Blob ? (I realize it might not have been available at that time).

Just create a Blob with your data, like var blob = new Blob([array1,array2,string,...]) and turn it back into an ArrayBuffer (if needed) using a FileReader (see this).

Check this : What's the difference between BlobBuilder and the new Blob constructor? And this : MDN Blob API

EDIT :

I wanted to compare the efficiency of these two methods (Blobs, and the method used in the question) and created a JSPerf : http://jsperf.com/appending-arraybuffers

Seems like using Blobs is slower (In fact, I guess it's the use of Filereader to read the Blob that takes the most time). So now you know ;) Maybe it would me more efficient when there are more than 2 ArrayBuffer (like reconstructing a file from its chunks).