Python concurrent.futures.wait job submission order not preserved

88 Views Asked by At

Does python's concurrent.futures.wait() preserve the order of job submission? I submitted two jobs to ThreadPoolExecutor as follows:

import concurrent.futures
import time, random

def mult(n):
    time.sleep(random.randrange(1,5))
    return f"mult: {n * 2}"

def divide(n):
    time.sleep(random.randrange(1,5))
    return f"divide: {n // 2}"


with concurrent.futures.ThreadPoolExecutor() as executor:
    mult_future = executor.submit(mult, 200)
    divide_future = executor.submit(divide, 200)

    # wait for both task to complete
    mult_task, divide_task = concurrent.futures.wait(
        [mult_future, divide_future],
        return_when=concurrent.futures.ALL_COMPLETED,
    ).done

    mult_result = mult_task.result()
    divide_result = divide_task.result()

    print(mult_result)
    print(divide_result)

Sometimes I see

divide: 50
mult: 400

and sometimes,

mult: 400
divide: 50

shouldn't mult_task, divide_task always map to mult_future, divide_future ?

python --version
>> Python 3.8.16
2

There are 2 best solutions below

2
Hai Vu On BEST ANSWER

According to the doc that the function wait():

Returns a named 2-tuple of sets.

By definition, sets are unordered. That means you cannot guarantee the order.

Since you already have mult_future and divide_future, you can use them to guarantee the order. There is no need to call wait either.

import concurrent.futures
import random
import time


def mult(n):
    time.sleep(random.randrange(1, 5))
    return f"mult: {n * 2}"


def divide(n):
    time.sleep(random.randrange(1, 5))
    return f"divide: {n // 2}"


with concurrent.futures.ThreadPoolExecutor() as executor:
    mult_future = executor.submit(mult, 200)
    divide_future = executor.submit(divide, 200)

# Print in order: mult div
print(mult_future.result())
print(divide_future.result())
0
Ahmed AEK On

futures.wait returns two sets, items in the set are ordered based on their address in memory because Future doesn't define a comparison operation, so the return order has nothing to do with the input order.

if you want to loop over outputs in input order then just loop over the input list.

in_vals = [mult_future, divide_future]
concurrent.futures.wait(
        in_vals,
        return_when=concurrent.futures.ALL_COMPLETED,
    ).done
for item in in_vals:
    ...