Recommendations for adding spot EC2 instances for short-term workload

Question

I have a php/Laravel application deployed to an AWS t2.small EC2 machine serving as my web server. 98% of the day, this little machine does just fine serving up HTTP and mySQL workload. However, I need to run a huge processing/batch/calc job for about an hour each day and was exploring how I might tap into Amazon's spot pricing to spin up a server each day for this daily workload.

My application uses queues to fire off different jobs and I was hoping that I could use the AWS SDK to automatically make a spot request then add it to a cluster to complete my queue processing.

I'm just a little unsure of the process I would go through to make this happen and thought I'd ask this community for any pointers or tutorials that show how one would tackle using Spot instances for this short-term workload processing?

Thanks in advance for any pointers or suggestions!

Answer 1

If your "huge processing/batch/calc" job can be interrupted, then Spot Instances could be good for you. However, if this job has to be completed before the next day and cannot be interrupted, I would avoid using Spot Instances.

When you request a Spot Instance, you put in a bid price. If your bid price is below the Spot price, you will not get a Spot Instance. Also if there is not enough unused capacity, then you will not be able to acquire a Spot Instance.

Your best bet will be to create and launch a larger On-Demand instance to perform this job each night.

Answer 2

Spot prices may vary by different AZ. Before requesting for spot, find which AZ has the best price. Loop through every few mins until one these conditions are met a) spot price is lower than your bid price in a subnet or b) reaches a threashold time. If you are unable to get a spot price until a threshold time, switch to on-demand and proceed.

Note that the threshold time should be very conservative. Spot instances are subject to fail (due to increase in prices) - so you should account into the extra time to switch to on-demand and still meet the time (SLA etc) to process.

Check spot price: http://docs.aws.amazon.com/cli/latest/reference/ec2/describe-spot-price-history.html

Request spot instance: http://docs.aws.amazon.com/cli/latest/reference/ec2/request-spot-instances.html