How do I upload a file to s3 using boto3 in python on heroku?

Question

All of the examples online seem to use boto. What code can I use to upload a file to s3 using boto3?

I'm trying to use the code from the official documentation, but I'm getting an error that version 4 authentication failed.

Answer 1

Just for reference here's how you can fix the issue of v4 authentication being used for all new regions - set signature_version in the config file:

--- .aws/config ---
[default]
output = json

[profile myprofile]
region = REGION_NAME
s3=
  signature_version = s3
#  addressing_style = path

--- .aws/credentials ---
[myprofile]
aws_access_key_id = <access-key>
aws_secret_access_key = <secret-key>

--- python ---
import boto3

session = boto3.Session(profile_name='myprofile')
s3 = session.resource('s3')
with open('PATH_TO_FILE_ON_DISK', 'rb') as data:
    s3.Object('BUCKET_NAME', 'FILENAME_ON_S3').put(Body=data)

Alternatively:

s3.Object('BUCKET_NAME', 'FILENAME_ON_S3').upload_file('PATH_TO_FILE_ON_DISK')

Answer 2

Put your keys in the keyfile according to the quickstart guide and then use the following code:

       import boto3
       s3 = boto3.resource(service_name='s3', region_name='REGION_NAME')
       data = open('PATH_TO_FILE_ON_DISK', 'rb')
       s3.Bucket('BUCKET_NAME').put_object(Key='FILENAME_ON_S3', Body=data)

The official documentation does not make it clear that the region name is required or else you may get an error, and it's not possible to store the configuration file for the region on heroku. It must be included in the resource call.

Answer 3

Since heroku doesn't have access to your AWS config file, you'll have to use environment variables. You can then use any of the various s3 file upload methods.

from os import getenv

import boto3

boto_kwargs = {
    "aws_access_key_id": getenv("AWS_ACCESS_KEY_ID"),
    "aws_secret_access_key": getenv("AWS_SECRET_ACCESS_KEY"),
    "region_name": getenv("AWS_REGION"),
}

s3_client = boto3.Session(**boto_kwargs).client("s3")

s3_client.upload_fileobj(<f>, <bucket_name>, <object_name>)

Answer 4

Recently, Heroku released a good article on uploading files to S3 in Heroku. Direct to S3 File Uploads in Python.

According to instructions, you need to specify AWS Credentials and Bucket name in your Heroku App config vars.

In order for your application to access the AWS credentials for signing upload requests, they will need to be added as configuration variables in Heroku:

$ heroku config:set AWS_ACCESS_KEY_ID =xxx AWS_SECRET_ACCESS_KEY =yyy
Adding config vars and restarting app... done, v21
AWS_ACCESS_KEY_ID     => xxx
AWS_SECRET_ACCESS_KEY => yyy

In addition to the AWS access credentials, set your target S3 bucket’s name:

heroku config:set S3_BUCKET = zzz
Adding config vars and restarting app... done, v21
S3_BUCKET     => zzz

Note: if you are testing locally before deployment, remember to add the credentials to your local machine’s environment, too. It was described above how to make it with a config file.