Python dynamic module loading with multiprocessing

Question

The following code works if the module "user.py" is in the same directory as the code, but fails if it is in a different directory. The error message I get is "ModuleNotFoundError: No module named 'user'

import multiprocessing as mp
import imp

class test():
    def __init__(self,pool):

        pool.processes=1
        usermodel=imp.load_source('user','D:\\pool\\test\\user.py').userfun
         #file D:\\pool\\test\\user.py looks like this:
         #   def userfun():
         #      return 1

         vec=[]
         for i in range(10):
            vec.append([usermodel,i])

         pool.map(self.myfunc,vec)

    def myfunc(self,A):
        userfun=A[0]
        i=A[1]
        print (i,userfun())
        return

if __name__=='__main__':
    pool=mp.Pool()
    test(pool)

If the function myfunc is called without the pooled process the code is fine regardless of whether user.py is in the same directory of the main code or in \test. Why can't the pooled process find user.py in a separate directory? I have tried different methods such as modifying my path then import user, and importlib, all with the same results.

I am using windows 7 and python 3.6

Answer 1

multiprocessing tries to pretend it's just like threading, but the abstraction leaks like a sieve. One of the ways it leaks is that communicating with worker processes involves a lot of implicit pickling and data copying.

When you try to send usermodel to a worker, multiprocessing implicitly pickles it and tries to have the worker unpickle the pickle. Functions are pickled by recording the module name and function name, so the worker just thinks it's supposed to do from user import userfun to access userfun. It doesn't know that user needs to be loaded with imp.load_source from a specific filesystem location, so it can't reconstruct usermodel.

The way this problem manifests is OS-dependent, because if multiprocessing uses the fork start method, the workers inherit the user module from the master process. fork is default on Unix, but unavailable on Windows.