RQ Worker Processing Jobs in a Batch

Question

Say you have a RQ Queue with lots of jobs, which gets filled from various sources.
Those Jobs would be more efficiently processed in batches, eg pulling and processing 100 Jobs at a time.

How would you achieve this in RQ?
Would you need to write a custom Worker class to pull multiple jobs at once, or a custom Queue Class to batch jobs when they are given out, or some other approach?

Thanks

Answer 1

I think that Tasktiger, which is based on redis like RQ, can better fulfill your needs. From the README:

Batch queues

Batch queues can be used to combine multiple queued tasks into one. That way, your task function can process multiple sets of arguments at the same time, which can improve performance. The batch size is configurable.