I have a technical issue running my Spring batch jobs. The Job simply reading records from the DB (MongoDB) , making some calculations on the record (aggregations) and writing the record result to another table. Reading A , Processing A , writing to record B B is an aggregations of many records of A. I want to use remote chunking to vertically scaling my system causing the processing part be scaled and quick. The problem I face that I need to synchronize the A records so processing them will not conflict when writing the result to B. If I distribute 10 A records to 4 slaves they will conflict when writing the aggregate result to B .
Any idea , how to add synchronizing policy when sending messages from the master to the slaves ?
Thanks in advance ...
If you need to synchronize data like you're describing, I'd recommend not going with remote chunking and using partitioning instead. This would allow you to partition by A and eliminate the synchronization issues you're facing. It would also provide additional throughput as you'd be running one processor per slave (same as in remote chunking).