Simultaneously Call a Script from Python, passing data to each, and Returning data to the original

Question

I am trying to create a python script that calls 5 other python scripts to run simultaneously while passing in an array and then each of the 5 scripts perform an operation on that array and return a different array to the initial script.

The initial script then realize when the 5 have returned values and then performs operations on these 5 arrays.

I think the solution is something like os.system(./script1.py arg1), os.system(./script2.py arg2) but I'm unsure of how to proceed.

Answer 1

You can use a thread pool to run all of the commands in parallel. I also changed over to the subprocess module which grabs program outputs:

import multiprocessing.pool
import subprocess as subp

def worker(script):
    proc = subp.Popen(script, shell=True, stdout=subp.PIPE, stderr=subp.PIPE)
    out, err = proc.communicate()
    return script, out, err, proc.returncode

scripts = ['./script1.py arg1', './script2.py arg2']
pool = multiprocessing.pool.ThreadPool(len(scripts))
for script, out, err, returncode in pool.map(worker, scripts):
    do your magic
pool.close()
pool.join()