Number of PUT requests for S3 folder upload

Question

I am new to AWS. I am working with S3 PUT objects and trying to upload 10 files in a folder.

To be more specific, I am uploading the folder. How many PUT requests does it take to complete uploading the files? Will it be 10 PUT requests because 10 files are treated as 10 objects in S3 or 1 request as we upload the file?

Answer 1

Adding a more descriptive answer here as Amazon S3 offers a range of storage classes optimized for different use cases. Such as..

1. Storage Object size limit.
2. Storage duration
3. Cost structure
4. Lifecycle management
5. retrieval items.

When we talked about AMAZON S3, it replicates data across three or more Availability Zones within a region those span across a minimum 1 km or max 100 Km just to avoid cases such as natural disasters and to ensure fault tolerance, resiliency, and LLT.

Look at the picture below:

Amazon S3 offers you eight different storage classes, below are the ones ...

1. Standard (frequently accessed data, more than once a month with ms/access )
2. Intelligent Tiering(S3 Intelligent-Tiering delivers milliseconds latency and high throughput performance for frequently, infrequently, and rarely accessed data in the Frequent, Infrequent, and Archive Instant Access tiers. )
3. Standard-IA (Infrequently accessed data, once a month with ms/access).
4. One-Zone IA (Recreatable, Infrequently accessed data, once a month, stored in Single AZ with ms/access)
5. Glacier Instant Retrieval (Long archived data once in a qtr with instant retrieval in ms)
6. Glacier Flexible Retrieval(Formerly Glacier): Long archived data once in a year with retrieval which can span from mins to hours depending on the data.
7. Glacier Deep Archive: Long archived data less than once a year with the retrieval of hours.

Now back to your original ask about AWS The PUT request

Precisely: When you are uploading a folder to s3 via API there you upload an object of up to 5 GB in a single operation saying that you normally require one put request for one file and that file may be upto 5GB max otherwise it will be multi-parted which further will require more put operations indeed.

Amazon S3 is a distributed system, The AWS S3 PUT request operation is used to add an object to a bucket.

To keep in mind, uploading a file to Amazon S3, it is stored as an S3 object. Objects consist of the file data and metadata that describes the object. Moreover s3 provides you freedom to upload any file type into a S3 bucket such as:

images

backups

data

movies

Blockquote

Another things to know while uploading a file to s3 bucket, there is a limit to upload a file:

• Via s3 console the upload is 160GB Max.
• For bigger files than 160GB you need to use either AWS CLI or AWS SDK or Rest API.

Depending on the size of the data you are uploading, Amazon S3 offers the following options:

• Upload an object in a single operation using the AWS SDKs, REST API, or AWS CLI—With a single PUT operation, you can upload a single object up to 5 GB in size.

• Upload a single object using the Amazon S3 Console—With the Amazon S3 Console, you can upload a single object up to 160 GB in size.

• Upload an object in parts using the AWS SDKs, REST API, or AWS CLI—Using the multipart upload API, you can upload a single large object, up to 5 TB in size.

• Multipart upload— The multipart upload API is designed to improve the upload experience for larger objects. You can upload an object in parts. These object parts can be uploaded independently, in any order, and in parallel. You can use a multipart upload for objects from 5 MB to 5 TB in size.

Using multipart upload provides the following advantages:

Improved throughput – You can upload parts in parallel to improve throughput.

Quick recovery from any network issues – Smaller part size minimizes the impact of restarting a failed upload due to a network error.

Pause and resume object uploads – You can upload object parts over time. After you initiate a multipart upload, there is no expiry; you must explicitly complete or stop the multipart upload.

Begin an upload before you know the final object size – You can upload an object as you are creating it.

How upload works when uploading a Dierctory/Folder:

When uploading files from a directory, you don't specify the key names for the resulting objects. Amazon S3 constructs the key names using the original file path. For example, assume that you have a directory called c:\myfolder with the following structure:

C:\myfolder
      \a.txt
      \b.pdf
      \media\               
             An.mp3

More details visit to AWS PUT KB .

• Amazon S3 does not support object locking for concurrent writers. If two PUT requests are simultaneously made to the same key, the request with the latest timestamp wins.

Look at the aws document about upload a Directory