I want to upload a 20 gigabyte file to S3, and calculate a sha256 checksum using the crypto api. Currently, I can do this in two promises chained together using two Readable Streams, but I'd like to optimize so that I can use only one Readable Stream. Looking at the AWS SDK implementation, the fillStream method uses the ReadableStream.read method. But it seems to me the way to send chunks of data to two consumers is to attach an event handler to the 'data' event.
According to ReadableStream there are two modes, paused and flowing. So what would attaching an event handler do? Would it skip chunks of data in either my reads or AWS SDK's reads? Would it trigger reading the same data twice from disk?
Are there other approaches to use the crypto api and the aws-ask package to optimize disk access?
You can either try to pipe one readable stream to two writable streams at the same time, or you can write a custom code that listens on data events and write to two streams.
There are also some modules on npm to make one stream split into two: