Trigger AWS Cloud9 to run when file is uploaded to a S3 Bucket

Question

I have a .py file with code in a AWS Cloud9 environment.

I want to run this code when a file is uploaded to a specific S3 Bucket.

But, it seems that I only can have triggers with AWS Lambda functions. I don't know how to add triggers to the AWS Cloud9 functions...

My simple code in the .py file in the Cloud9 environment is:

import boto3
import os
import sys

s3 = boto3.client('s3')

s3.download_file('our-awesome-first-test-bucket', 'test_text.txt', 'test_text_saved_to_env.txt')

with open('test_text_saved_to_env.txt', 'r') as f:
    output = sum(map(int, f))
   
with open('output_to_awesome_bucket.txt', 'w') as outf:
    outf.write(str(output))
   
s3.upload_file('output_to_awesome_bucket.txt', 'outputbucket-for-our-first-awesome-bucket', 'output_to_awesome_bucket.txt')

os.remove('test_text_saved_to_env.txt')
os.remove('output_to_awesome_bucket.txt')

How can I trigger this code to run when a file is uploaded to the S3 Bucket?

Answer 1

As mentioned in the documentation, S3 supports the following destinations as a trigger once an object has been uploaded:

• Amazon Simple Notification Service (Amazon SNS) topics
• Amazon Simple Queue Service (Amazon SQS) queues
• AWS Lambda function

The ideal solution to use would be to simply run a lambda function, but as you mentioned you have dependencies on libraries which have a size greater than 250MB, which is the limit for lambda.

In this case, you should consider packaging up your Python application in a Docker image which you can run in an ECS task. You should also create a lambda function as a wrapper which will invoke the ECS task.

Configure the bucket to trigger the lambda which, in turn, will run the ECS task.

---

As an alternative, if you wish to run the script on the Cloud9 EC2 instance, you can configure a service on the EC2 instance which polls an SQS queue (configured as a target for your S3 bucket) and then runs your script if an event has been detected in the queue.