use boto3 session when opening s3 url

800 Views Asked by At

I see a lot of code that uses an S3 bucket url to open a file. I would like to use smart open to open a compressed file

session = boto3.Session(ID, pass)
file = open("s3://bucket/file.txt.gz", transport_params=dict(session=session), encoding="utf-8")

However, all the examples I see about smart open and other pulls from boto3 using a url never specify how to use the session when pulling the data from the url only when pushing data to a new bucket. Is there a way to use the url and the session without needing to create a client from my session and access the bucket and key?

1

There are 1 best solutions below

0
On BEST ANSWER

As mentioned you just need to replace "wb" with "rb". I was mistaken that this didn't work

from smart_open import open
import boto3

url = 's3://bucket/your/keyz'
session = boto3.Session(aws_access_key_id,
                        aws_secret_access_key,
                        region_name)

with open(url, 'rb', transport_params={'client': session.client('s3')}) as fin:
    file = fin.read()

print(file)