I have some tasks which I want to execute in parallel in Java. I want to generate the next task to perform myself, based on data I'm gathering "in real time" (mostly, it depends on which other tasks have finished and what their results were).
Ideally I'm looking for an executor constructed like so:
new CallbackThreadPoolExecutor(Iterator<Callable<T>> generator)
where I imagine the generator has some internal state recording information about how the execution is progressing, and based on this information, whenever next() is invoked, it generates the next task.
Note a ThreadPoolExecutor requires to queue the tasks, but I want to use the most up-to-date information at the time when the decision has to be made.
Here's a clunky workaround: create a ThreadPoolExecutor with nThreads tasks, each of which will call the generator in a loop. But is there a more idiomatic / efficient way of doing this in Java? (perhaps using a ForkJoinPool?)
this question seems to address a related problem (a very large list of tasks) but here the problem is different: I want to make decisions on the fly - and you can assume the number of tasks is not too large (but I may not know the precise number of tasks in advance).
Typical Application: Suppose we are given a directed acyclic graph, whose vertices are tasks and whose edges represent dependencies. We can only perform a task if all its ancestors have completed. We want to use multiple workers to process all the tasks, but since we do not know precisely how long each task will take, we want to choose the next task to assign to a worker only when the worker becomes free, based on the most up-to-date state of the graph (ideally, we want to choose only tasks whose ancestors have completed; of those, perhaps we want to prioritize tasks which have many dependents).