Python: How to get the unordered result from starmap_async

Question

So the thing is I want to get the return value from the starmap_async. However if I use the .get() to get the list of the result, it is little slow and same as starmap. How do I change the following sample code?

full_res = []
def return_result(result):
    full_res.append(result)

with  mp.get_context("spawn").Pool(5,maxtasksperchild=1000) as pool:
            result=pool.starmap_async(fun, [(args)],callback=return_result)
        pool.close()
        pool.join()

By the way, I do not need the order.

Answer 1

If you don't care about the order of results and you want to get results as they become available, consider using the concurrent.futures module instead:

import concurrent.futures
import random
import time


def do_work(id):
    sleeptime = random.randint(0, 10)
    time.sleep(sleeptime)
    return id, sleeptime


with concurrent.futures.ProcessPoolExecutor() as executor:
    tasks = [executor.submit(do_work, (x,)) for x in range(10)]
    for task in concurrent.futures.as_completed(tasks):
        print(task.result())

Run this code, and you will see that results are displayed as soon as they become available. There is a map method available that may be appropriate depending on the nature of your work function; we could rewrite the above like:

import concurrent.futures
import random
import time


def do_work(id):
    sleeptime = random.randint(0, 10)
    time.sleep(sleeptime)
    return id, sleeptime


with concurrent.futures.ProcessPoolExecutor() as executor:
    tasks = executor.map(do_work, range(10))
    for result in tasks:
        print(result)