Debug forever pending Celery task calling OptaPy

Question

I try running OptaPy in a Celery task from within a Django application using Redis as a broker. The task is received by the Celery worker, but remains PENDING forever and returns no result.

If I set CELERY_TASK_ALWAYS_EAGER = True for debugging, the code runs fine, but synchronously.

What could be the issue? Which steps can I take to find out?

As OptaPy is a Python wrapper for a Java library, I suspect that Celery might not handle the additional thread well, but I don't know how to debug or fix this.

This is the Celery task in question, additional code can be provided on request:

@shared_task()
def schedule_task():

    solver_config = optapy.config.solver.SolverConfig() \
        .withEntityClasses(Lesson) \
        .withSolutionClass(TimeTable) \
        .withConstraintProviderClass(define_constraints) \
        .withTerminationSpentLimit(Duration.ofSeconds(30))

    solution = solver_factory_create(solver_config) \
        .buildSolver() \
        .solve(generate_problem())

    return str(solution)

Answer 1

This answer seems to apply here, too:

By default celery uses the multiprocessing library as the concurrency implementation. [...] Unfortunately threads don't inherit after a subprocess has been forked, so a zombie is created.

Starting the Celery worker with the --pool threads option does the trick for me:

celery -A myapp worker --pool threads --loglevel=info