I'd like to create a farm of processes that are able to OCR text. I've thought about using a single queue of messages which is read by multiple OCR processes.
I would like to ensure that:
- each message in queue is eventually processed
- the work is more or less equally distributed
- an image will be parsed only by one OCR process
- An OCR process won't get multiple messages at once (so that any other free OCR process can handle the message).
Is that possible to do using AMQP?
I'm planning to use python and rabbitmq
Yes, that's possible. Server cluster for a real-time MMO game I'm working on operate this way. We use ActiveMQ, but I think all this possible with RabbitMQ as well.
All items that you mentioned you get out of the box, except last one.
To make last one work in desired way, consumers send AMQ-specific argument when subscribing to the queue:
This setting guarantees that consumer will not take any more messages after it took one and until it send an
ack
to AMQ. I believe something similar exists in RabbitMQ.