Ensure data is not lost during AWS S3 sync

327 Views Asked by At

I need to copy all contents of an S3 bucket to another S3 bucket. Planning to use s3 sync.

aws s3 sync s3://sourcebucket s3://destinationbucket

After this process, is there any way to verify if all data is migrated to the new bucket? (i.e no data is missed or lost)

Or is there any guarantee that data will not be lost (specified anywhere in official doc?)?

1

There are 1 best solutions below

0
On

Assuming you want this verification done after the sync is terminated. S3 provides MD5 hash of objects as ETag. You can traverse through your local directory making sure that object does exists in the S3 bucket and integrity can be verified by comparing the local and remote MD5 hashes. (https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectGET.html#RESTObjectGET-responses)