What Microservices pattern is appropriate for transfering large data file

Question

If a large JASON file(10mb) is needed for processing by multiple Microservices what's the best Enterprise Architectural/Design pattern to use? Entire data in the file is needed by each Microservices in order to process it.

Answer 1

Store it somewhere where other microservices can read it from. Using pub/sub or eventing - notify interested microservices where the file is published. The other microservices can read it for themselves.

Answer 2

1. client/system POSTs an event to an API that implements a Saga pattern, passing a link to where the file is located (e.g. uploaded to an AWS S3 bucket).
2. Each event/step in the Saga pattern processes the file, per their requirements.
3. client/system can poll for an update, or the original POST can be designed to pass a callback/notification service endpoint.

Answer 3

• Sharing large data set may be an indication for a suboptimal partitioning of the codebase into services. It is preferred all processing of the same domain will be done within a single service.
• When multiple services do have meaningful processing to be done on the same data set - each should have its own copy of it. Sharing databases, is typically - a bad idea!
• When heavyweight data is involved, cloning the data in a "regular" queueing system (such as RabbitMQ / SQS) is quite cumbersome and inefficient.
  • A "heavyweight" queuing system such as Kafka / Kinesis - may be most efficient. One copy of the data will be persisted, and each service can read it from a "shared" stream.