I am using the multiprocessing module in Python to map a set of jobs onto as many cores as I have. The jobs I am wrapping are mostly performed with some wrapped fortran code (wrapped with f2py). When I call the jobs only, they go through fine using all cores. However, if I try to call the fortran library independently (not through the Pool().map() function) firstly, and then call the map, the whole program hangs (if you want to know, it hangs precisely at line 339 of threading.py on which the only code is waiter.acquire()).
For the program I am writing, I need to call the library before I go and map it. I have gotten around this by calling it firstly using Pool().map() with just a single process, but this is a hack, and I'd like to know why I am having to do this!
To be more explicit, this does not work:
def call_camb_transfer(H0): #wraps the call to define the keyword parameter to pass
return pycamb.transfers(H0=H0)[1][6, :, 0]
b = call_camb_transfer(70)
pool = multiprocessing.Pool(2)
H0 = [60, 70]
results = pool.map(call_camb_transfer, H0)
but this does:
def call_camb_transfer(H0): #wraps the call to define the keyword parameter to pass
return pycamb.transfers(H0=H0)[1][6, :, 0]
initial_pool = multiprocessing.Pool(1)
b = initial_pool.map(call_camb_transfer, [65.0])
pool = multiprocessing.Pool(2)
H0 = [60, 70]
results = pool.map(call_camb_transfer, H0)
where pycamb is the python wrapper of CAMB, which is a fortran library.