Error handling while using chain of groups in celery

Question

I have 100 tasks. But I want to process only 4 tasks at a time. Once these 4 tasks get completed i want to run the next set of 4 tasks. This can be done by grouping tasks in sets of 4 and then chaining them. But some tasks may fail in between, how can I handle these errors and take appropriate actions, so that I retry for only the tasks that failed. As per my knowledge in a chain if one task fails, the subsequent tasks in the chain will not run. So if I implement chain of groups, if any tasks in a group fails, the entire chain will fail. Suggest me a proper error handling method for this, also any better idea for implementing this will be appreciated.

I have disabled the result backend.

A little background - I have a cron running every 30s implemented using celery-beat which picks up 100 new tasks at a time. So all these chaining and grouping has to be done inside the cron function.

I am using celery 3.1.25

Answer 1

Don't chain the tasks together if they are independent. Adjust the settings on the celery worker for a specific queue to limit the number of workers to 4 using --concurrency 4 or -c 4.