I am designing a NodeJS app that downloads, parses, and uses basic machine learning techniques on publicly available data from a handful of different sources. This can be computationally expensive, and in order to avoid flooding the main event loop of my app, I would like to make use of child processes / microservices to isolate each major task into its own process.
Ideally, the process segregation would look something like this:
Main Process
|-Data Source A
| |- Data Downloading Microservice
| |- Data Parsing Microservice
| |- Main Computation Microservice
|-Data Source B
| |- Data Downloading Microservice
| |- Data Parsing Microservice
| |- Main Computation Microservice
...
From what I currently understand, data piping and communication between each of these processes would be limited, and when the time comes to access a specific piece of information from one of the Data Source child processes, I would have to send a message from the Main Process to the Data Source process, and then have the Data Source process send a message to one of its microservices, and then pass that data all the way back up.
Is there an easier way to do this? For instance, being able to store a table of process ids for each sub process in a lookup table and be able to directly send messages to and from that process without going through the one that initially spawned it?