How can I forward a mulitpart file stream from fastify-multipart to a 3rd part api?

Question

I have a node api that I'm building that's capable of handling uploads of multipart file types (chunks). This api is based on the Fastify library, and I've already installed the separate Fastify-Multipart library. I've got everything working, including mulitpart file uploads, but part of the requirements of this api is to be able to send requests to another api. In particular, I need to send the file uploads. I don't know what their api is written in, but their multipart file upload api is basically like this:

sdk.files.uploader(location_id, file_size, "filename.jpg", file)
.then(uploader => uploader.start())
.then(file => { console.log(file) })

My code is basically this:

post: async (request, reply) => {

        // Check for file upload
        if((request.headers['content-type'] && request.headers['content-type'].indexOf('multipart/form-data') !== -1) && request.params.id) {

            const multipart = request.multipart(async (field, stream, filename, encoding, mimetype) => {

                console.log(`folderId: ${request.params.id} filename: ${filename}, 
                            field name: ${field}, encoding: ${encoding}, 
                            mime type: ${mimetype}, file length: ${request.headers['content-length']}`)

                try {
                    let uploader = await sdk.files.uploader(request.params.id, Number(request.headers['content-length']), filename, stream)
                    let file = await uploader.start()
                    console.log(file) //Never reaches this point
                }
                catch(e) {
                    console.log(`An error occurred during upload: ${e.message}`)
                    reply.code(500).send()
                }
                //pump(file, fs.createWriteStream(filename))

            }, (error) => {

                if(error) {
                    console.log(`Error uploading file: ${error.message}`)
                    reply.code(500).send()
                } else {
                    console.log('File upload succeeded') //Upload succeeds but it's just in memory
                    reply.code(201).send()
                }
            })

            multipart.on('field', (key, value) => {
                console.log('form-data', key, value)
            })
        }
    }

So basically what I want to do is pass a multipart file stream to this 3rd party api, but doing it this way doesn't seem to be working (when I go on their site, I don't see the file in the folder where it should be). When I look at Activity Monitor on my machine (macOS), I see the node process is consuming 1.2 Gig of memory (roughly the size of the file). Does anyone know a way to do this using Fastify-Multipart (which I believe is based on BusBoy).

Answer 1

I notice that your handler post: async (request, reply) => is async, but you are not calling await, but you are managing the reply in the multipart callback. This can cause issues. Read the promise resolution doc for details.

I would suggest to check the module you are piping the stream since it must use the steam approach and don't save all the chunks to memory.

Here a simple example:

const fastify = require('fastify')({ logger: true })
const pump = require('pump')

fastify.register(require('fastify-multipart'))

fastify.post('/', function (req, reply) { // this function must not be async
  if (!req.isMultipart()) { // you can use this decorator instead of checking headers
    reply.code(400).send(new Error('Request is not multipart'))
    return
  }

  const mp = req.multipart(handler, onEnd)

  mp.on('field', function (key, value) {
    console.log('form-data', key, value)
  })

  function onEnd (err) {
    if (err) {
      reply.send(err)
      return
    }
    console.log('upload completed')
    reply.code(200).send()
  }

  async function handler (field, file, filename, encoding, mimetype) {
    console.log('.....waiting')
    await wait() // testing async function
    pump(file, ConsoleWriter({ highWaterMark: 1000 }))
  }
})

fastify.listen(3000)

function wait () {
  return new Promise(resolve => {
    setTimeout(resolve, 1000)
  })
}

// A writer that manage the bytes
const { Writable } = require('stream')
function ConsoleWriter (opts) {
  return new Writable({
    ...opts,
    write (chunk, encoding, done) {
      console.log({ chunk: chunk.length, encoding })
      setTimeout(done, 500) // slow simulation
    }
  })
}

Call it with:

curl -F file=@"./README.md" -H 'content-type: multipart/form-data' -X POST http://localhost:3000/