I am working on a Django application where users can upload files to be processed by Celery tasks. Each file upload triggers a Celery task that processes the file individually. The problem arises when a user uploads a large batch of files (e.g., 10,000 files), as it causes subsequent users' uploads (e.g., 100 files) to wait until the first user's batch has completely processed. This results in significant delays and a poor user experience for those uploading smaller batches of files.
Tasks are segregated into user-specific queues.
Each queue is named following the pattern user-{user_id} (e.g., user-1, user-2, ..., user-10).
My objective is to achieve a round-robin processing strategy across these queues: after processing a task from user-1's queue, the worker should move on to process a task from user-2's queue, and so on, cycling through the queues in a round-robin fashion until all tasks are processed.
Stuck on:
I have not been able to configure Celery to listen to queues using a pattern (e.g., user-.*). The queue list for the Celery worker must be predefined and cannot dynamically adjust to match new user-specific queues as they are created.
Question:
Is there a way to configure Celery workers to process tasks in a round-robin manner across multiple user-specific queues?