We've a set of microservices for document processing. One of the service will generate a document from template, another will convert to PDF and another one will save to database and so on. The microservices communicates through messages using SNS/SQS. Currently we are storing the temporary files / data generated by microservice in S3 since we cant send them as part of messages. There is a time/cost penalty in reading and writing these documents to S3 and also we want to keep them for a short time. Is S3 is the best option to store these kind of temporary data or is there any cache mechanism like Elastic cache we can leverage?
The files are like word, pdf docs size ranging from KBs to MBs