Multiprocessing in python compatible from 3 to 2.7

Question

I have the following code runnning smoothly in Python 3, and I can't convert it to Python 2.7.

from multiprocessing import *

def func(path, filename, path_2):
    #Things to do

for i in range(0,len(chr_names)): #len(chr_names) = 24
    tuple_var.append((path, chr_names[i][0], chrom_sizes[i][0]))

cores = 4
with Pool(cores) as p:
    p.starmap(func, tuple_var)

I get the following error.

python AttributeError: __exit__

I know starmap is not supported in Python 2.7.

What code should I use in Python 2.7?

Answer 1

One simple approach, use a wrapper function:

def star_wrapper(args):
    return func(*args)

....

with Pool(cores) as p:
    p.map(star_wrapper, tuple_var)

Answer 2

Unless I'm misunderstanding you, it seems like you can use Pool's map function in Python 2.6+. All you need is a function that can apply tuple arguments to the original function. Something like:

def pool_starmap(pool, fn, items):
    def map_fn(args):
        fn(*args)
    return pool.map(map_fn, items)

cores = 4
with Pool(cores) as p:
    pool_starmap(p, func, tuple_var)

Answer 3

The other answers have already covered how you can port starmap, but as for the AttributeError: __exit__ error, it comes from the fact that multiprocessing.Pool cannot be used as a context manager in Python 2.7, so you would simply have to do the following instead:

p = Pool(cores)

Answer 4

First:

In Python 2.x and 3.0, 3.1 and 3.2, multiprocessing.Pool() objects are not context managers

Have a look at this post for more info: Python Multiprocessing Lib Error (AttributeError: __exit__)

Second:

Use a helper function

Or choose one of the other options presented here: Multiprocessing with multiple arguments to function in Python 2.7

Sample code:

from contextlib import contextmanager
from multiprocessing import *

@contextmanager
def terminating(thing):
    try:
        yield thing
    finally:
        thing.terminate()

def func(path, filename, path_2):
    # Things to do
    print(path)
    print(filename)
    print(path_2)
    print('-----------------------------\n')

def helper(args):
    return func(args[0], args[1], args[2])

def task():
    tuple_var = []
    for i in range(0, 10):
        tuple_var.append(('hi_' + str(i), i, i))

    with terminating(Pool(processes=2)) as p:
        p.map(helper, tuple_var)

if __name__ == '__main__':
    task()