Upload a file from URL to another server

Question

I'm trying to upload a public video file from Amazon S3 to Dailymotion using a NodeJS Lambda function. AWS Lambda functions have a storage limitation: I can't download the whole video file locally on the serverless function and then send it to Dailymotion.

I searched for a way to read the file on parts and send them with HTTPS requests to Dailymotion, and I found the urllib library that seems helpful!

To upload a video file to Dailymotion, you can send a POST request to an UPLOAD_URL that should be generated using another authenticated request:

curl -X POST \
     -F 'file=@/path/to/your/video.mp4' \
     '<UPLOAD_URL>'

The steps are clearly mentioned on this document page.

What I need is a way to stream the file from a URL like this https://cdn.example.com/videos/filename.mp4 to Dailymotion through my Lambda function.

Anyone with good experience can help with a clear example of how to do it.

Answer 1

I found a solution using request and form-data libraries. Just:

1. Created a new form.
2. Appended the file to the form as a stream using request.
3. Submit the form.
4. Wait for the upload to end to send the callback back to the caller with the response.

Here's a working copy of my Lambda function code:

import request from 'request';
import FormData from 'form-data';

export const handler = (event, context, callback) => {
  const form = new FormData();
  form.append('file', request('https://cdn.mysite.com/video.mp4'));
  form.submit({
    host: 'upload-XX.dcX.dailymotion.com',
    path: '/upload?uuid=XXXXX&seal=XXXXX&extra=XXXXX'
  }, (error, response) => {
    if (error) throw error;
    let body = '';
    response.on('data', chunk => {body += chunk.toString()});
    response.on('end', () => {callback(null, JSON.parse(body))});
    response.resume();
  });
};

Update

Another solution using the https native module of Nodejs instead of the deprecated request library:

import https from 'https';
import FormData from 'form-data';

export const handler = (event, context, callback) => {
  const form = new FormData();
  https.get('https://cdn.mysite.com/video.mp4', response => {
    form.append('file', response);
    form.submit(
      {
        host: 'upload-XX.dcX.dailymotion.com',
        path: '/upload?uuid=XXXXX&seal=XXXXX&extra=XXXXX'
      },
      (error, response) => {
        if (error) callback(error);
        let body = '';
        response.on('data', chunk => body += chunk.toString());
        response.on('end', () => callback(null, JSON.parse(body)));
        response.resume();
      }
    );
  });
};