Need help designing video transcoding processing using EC2?

Question

I'm currently transcoding user uploaded videos in a linode server and using s3 to store them, but, being the optimist I am, I want to move the transcoding to Amazon EC2 to scale up if necessary, also to gain experience from using it. This is my work flow so far:

1. Upload videos directly to s3, bypassing nginx/Rails stack
2. Create an entry in Amazon simple queue service with video url
3. See if any instances of EC2 are running, if not start one with a simple ruby script as startup script (I believe Amazon allows startup scripts under 6kb)
4. Ruby script connects to simple queue and runs the ffmpeg command to transcode the video
5. Script sees if there are any additional queues and runs them
6. Shutdowns the instance when there are no more queues (might make it run the full hour since Amazon charges for full hour for any part of the hour EC instance is running)

Obviously the above is basic and doesn't use EC2 to its full purpose. I've thought about using threads to connect to queue and run new jobs on same EC2 instance or create additional EC2 instances, where the startup script will run the ruby script. With the former we'll need to limit number of jobs in same EC2 based on cpu usage.

The latter seems a waste, but given video transcoding is cpu intensive, maybe two ffmpegs at same time is not feasible. I've also thought about Amazon's auto scaling to create new instances, but using Ruby seems simpler and easier to me.

1. My question is what is the best way to do this efficiently?
2. Is running two ffmpegs considered bad practice? (Let's assume the video size will be around 200mbs average.)
3. Is using Ruby threads optimal given the bad name they seem to have?
4. Should I look into EventMachine in place of threads?

I don't want to run any EC2 instances unless necessary, get the maximum juice out of the instances, but not let my users wait too long for their videos to be transcoded.

Based on this article http://stream0.org/2009/10/h264-video-encoding-on-amazons.htmlhttp://stream0.org/2009/10/h264-video-encoding-on-amazons.html High-CPU, Extra Large Instance seems to be the best option. Of course I plan to do some of my own testing, but I wanted to get some expert opinion before I dive in. Thanks!

This turned out to be an essay, sorry for the length.

Answer 1

I ran some tests today on my local machine to test CPU usage with multiple ffmpeg processes. I found the following command on the Internet and so far it works decently, it encodes into flv, reduces file sizes without noticeable difference in quality. I know next to nothing about ffmpeg so maybe the command is total crap (please let me know if it is). One problem is it doesn't support threads in ffmpeg, but I think that might be a codec thing.

ffmpeg -i explanation-ace-2-3.mp4 -ab 96k -ar 22050 -qscale 6 output3.flv

I used top -b -d 0.5 in half second intervals to measure CPU usage and did grep Cpu to get relevant info. The files were about 150mbs in size and were encoded with the same ffmpeg command. I let the process run a little before starting a new one and here are my results:

1 FFMPEG | **0.84mbs/sec**
- 306.31 secs - 5:06 

Cpu(s): 23.6%us,  2.9%sy, 73.6%id  
Cpu(s): 23.7%us,  2.4%sy, 73.9%id  
Cpu(s): 23.6%us,  1.9%sy, 74.5%id  

2 FFMPEG | 1.6 mbs/sec
- 319.25 secs - 5:18
- 319.04 secs - 5:18

Cpu(s): 45.9%us,  4.4%sy, 49.8%id  
Cpu(s): 45.0%us,  5.7%sy, 49.3%id  
Cpu(s): 42.9%us,  8.3%sy, 48.8%id  

3 FFMPEGs | **2.23 mbs/sec**
- 344.72 secs - 5:44
- 342.12 secs - 5:42
- 340.39 secs - 5:40

Cpu(s): 68.8%us,  7.3%sy, 23.9%id  
Cpu(s): 70.9%us,  6.9%sy, 22.2%id  
Cpu(s): 66.0%us,  9.9%sy, 24.1%id  

4 FFMPEGs | **2.69 mbs/sec**
- 379.36 secs - 6.32
- 379.70 secs - 6.33
- 380.89 secs - 6.35
- 378.14 secs - 6.30

Cpu(s): 88.2%us, 11.8%sy,  0.0%id  
Cpu(s): 87.7%us, 12.3%sy,  0.0%id  
Cpu(s): 89.7%us, 10.3%sy,  0.0%id

5 FFMPEGs | **2.61 mbs/sec**
- 459.79 secs - 7.66
- 469.03 secs - 7.82
- 448.15 secs - 7.47
- 458.62 secs - 7.64
- 489.47 secs - 8.16

Cpu(s): 93.1%us,  6.4%sy,  0.0%id
Cpu(s): 88.7%us, 11.3%sy,  0.0%id
Cpu(s): 92.2%us,  7.8%sy,  0.0%id

Based on the data, converting videos one by one is gross under usage of resources, also ffmpeg processes are pretty stable, minus couple spikes. 4 instances of ffmpeg, at least for my machine, seems to be most efficient.

I seem to be having trouble running two ffmpeg process parallel using Thread or fork using system command.

Does anyone have any thoughts on this? Especially how to run two ffmpeg process using a Ruby script?