Multiprocessing.pool.Pool on Windows: CPU limit of 63?

Question

When using python's multiprocessing.pool.Pool with more than 63 cores, I get a ValueError:

from multiprocessing.pool import Pool

def f(x):
    return x

if __name__ == '__main__':
    with Pool(70) as pool:
        arr = list(range(70))
        a = pool.map(f, arr)
        print(a)

Output:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\fischsam\Anaconda3\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Users\fischsam\Anaconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\fischsam\Anaconda3\lib\multiprocessing\pool.py", line 519, in _handle_workers
    cls._wait_for_updates(current_sentinels, change_notifier)
  File "C:\Users\fischsam\Anaconda3\lib\multiprocessing\pool.py", line 499, in _wait_for_updates
    wait(sentinels, timeout=timeout)
  File "C:\Users\fischsam\Anaconda3\lib\multiprocessing\connection.py", line 879, in wait
    ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
  File "C:\Users\fischsam\Anaconda3\lib\multiprocessing\connection.py", line 811, in _exhaustive_wait
    res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 72
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69]

The program seems to work fine; the result is just what I expected. Can I ignore the ValueError?

Background: I have googled this issue, and it seems having to do with limitations of python on Windows (_winapi.WaitForMultipleObjects); see e.g. here. The suggested fix is to limit the number of used cores to 63. This is not satisfactory, because I want to 100+ cores on my server. Do I really need to limit the cores? Why? Is there a workaround?

Answer 1

I have written a package ("ObjectProxyPool") to do object-based parallelism with any number of parallel processes.

(Object-based parallelism: You have a copy of an object in each parallel process and can call its methods without having to re-initialize the object.)

The package is available on PyPi. You can use the tool to do stuff similar to classical multiprocessing.

import os
from objectproxypool import ProxyPool
import numpy as np

# Define a dummy class containing only the function 
# you are interested in
class MyClass:
    
    # Implement the function you are interested in.
    # Static methods are not supported.
    # You may just ignore the self argument.
    def f(self, x):
        return os.getpid()

if __name__ == '__main__':
    
    # Create a proxy pool with 70 workers in independent processes. 
    # If `separateProcesses = False`, the workers will be threads.
    with ProxyPool(MyClass, numWorkers=70, separateProcesses=True) as myObject:
        
        # myObject is a proxy to the object of type MyClass.
        # `map_args = True` assures that each worker gets one argument at a time.
        # Otherwise, each worker will receive the argument `range(100)`
        a = myObject.f(range(100), map_args=True)
        
    print("We used {} different processes.".format(np.unique(a).size))
    print(a)

Answer 2

Interestingly, when I went to see if the problem still existed using PrcocessPoolExecutor(max_workers=70) from the concurrent.futures module, I get ValueError: max_workers must be <= 61 and the program terminates immediately before any jobs can be submitted. This strongly suggests a Windows limitation that cannot be circumvented. Your program, however, never actually terminates and just hangs after getting the exception. On my computer with 8 cores, it hangs if I specify any number greater than 60 (not 61 nor 63) workers regardless of whether I use multiprocessing.Pool or concurrent.futures.ProcessPoolExecutor. Find which is the max number of workers on your machine that allows it to terminate normally without generating an exception and stick to it.

from concurrent.futures import ProcessPoolExecutor

def f(x):
    return x

if __name__ == '__main__':
    with ProcessPoolExecutor(max_workers=70) as executor:
        a = list(executor.map(f, range(70)))
        print(a)