Using SQS for batch running

Question

I am starting a new architecture to support a MDM (Master data management) database. For the MDM database I will be using a graph database (Neo4J). So the I need to integrate disparate sources within the MDM database. I will have sources from orders, customers signup, likes from facebook and so on.

So I imagine that I can have one or multiple SQS queues for the sources. So each source will put messages on SQS.

Then I will have many worker nodes that will be responsible to get the messages from queue and update the MDM database.

At the moment, I will use just one worker, because I dont know how neo4j will performance doing multiple writers.

So I see many people using SQS to send messages to launch a batch process. Its like a job queueing.

Is it a common use case use SQS to send data messages? Or maybe using another AWS component.

Answer 1

Using Neo4j with a queueing system is pretty common. I've never used SQS myself, but being aware of people using ActiveMQ (or others) to feed data into Neo4j.

This is mostly done do decouple the architecture and provide a way to gracefully deal with load peaks.

Depending on the kind of events you put onto the queue it might be beneficial if the consumer is single threaded - esp. when the events write the same part of the graph. Otherwise Neo4j's locking mechanism will prevent concurrent writes onto the same nodes/rels.