I have a mongodb collections which are 20 in number that i am using to store some data regarding a tasks that i am currently processing using cron jobs.
I have one worker per collection when using cron jobs. I want to improve this arrangement and i am looking into celery. I want to have at least 4 workers per collection since i have many records in each collection.
I want the jobs to be done as they come and not wait for the five minutes wait as its happening when using cron jobs.
Is this possible for me to have 4 workers per collection in celery in the way i have described?.
Celery workers will pick tasks as soon as a new task is initiated and will execute it, celery can use redis, or rabittMQ for storing the tasks queue. Any day you can scale the celery by running it distributed on multiple machines or by scaling up the machine and increasing the number of workers. https://www.slideshare.net/nicolasgrasset/scaling-up-task-processing-with-celery
Instead of using the crontab, use celery beat which is the task scheduler for celery.
There is no need of having collection wise celery workers.
Please go through the below celery documentation for understanding celery.
http://docs.celeryproject.org/en/latest/getting-started/introduction.html