How is object different from a file?

Question

How do Walrus, S3 or any cloud storage system, take in a file and convert it to object programmatically?

Answer 1

If you're asking how does S3 lay its objects out on disk on its internal servers, we don't know (the implementation details are not public.)

What they are very likely doing is taking your object's key (bucket + path) and consulting a consistent hash that maps your object to a set of servers. The upload is directed to one of these servers (essentially at random) which stores it and enqueues future work to propagate the new object to the other servers responsible for replicating it.

This replication delay is the underlying need for eventual consistency. I also heard once somewhere that Amazon uses an error-correcting encoding at the storage level to further defray bad reads.

Answer 2

To get the general understanding of how Cloud Object Storage systems stores the objects(Binary Files), you can read the documentation of Swift Object Storage-Openstack.

Swift is similar to Amazon S3 and hence to Walrus.

Swift communicates through the Proxy Server to Clients(Outside of the Cluster) and Client can store,delete the objects through the RESTful HTTP API. The server maintains the Ring-Configuration file that maintains the mapping between files and their physical location.

When there is a request to upload a file, MD5 hash is calculated from the path of the file.

Please find the details at:

Openstack Swift Architecture

Swift Documentation