I have celery setup to fetch tasks from RabbitMQ and things are working as excepted, but I've noticed the following behavior (T: task, P: process):
--> Fetch first batch of messages (6 tasks) from broker
<-- messages are received. Start them
--> Send T1..T6 to be executed by P1..P6
--> Prefetch 6 new messages from broker, but do not ACK them
<-- P1..P5 finish tasks T1..T5, but T6 is still being processed (it will take ~2h)
At this point, no other tasks start running, despite the fact that I have concurrency set to 6 and only one process is active. I have tried the add_consumer command on celery-flower, but nothing seems to happen. I can see on RabbitMQ that there are messages with no ACK yet and the messages in the READY state just start stacking up, since they won't be consumed for another ~2h.
Is there a way to setup celery so that whenever a process is free, it will consume the next task, instead of waiting for the original batch to completely finish?