Can the cluster_status callback submit new jobs?

Question

I'm developing a dispy script operating on a large number of objects (duh).

There are multiple operations to be performed for each object -- only one of them must happen first, the rest use the first job's results and can be done in any order.

cluster = dispy.JobCluster(
    getShift,
    setup = functools.partial(doLoads, from, to),
    cluster_status = processed,
    nodes = Nodes
)

The straightforward way, I suppose, is to do two loops of cluster.submit() -- first loop for the first job for each object, and the second loop -- for all others.

However, I'd like to begin submitting the subsequent jobs as soon as the primary one is finished for each object -- without waiting for it to be done for all the other objects.

What's the right way to do that? In particular, can my processed callback function submit additional jobs to the same cluster?

Answer 1

Yes, callback can submit new job(s). See, for example, job_scheduler.py, which submits jobs in callback (although cluster_status can also be used for this purpose). From above description, it seems you can use callback instead of cluster_status.