how to aggregate the result in different subprocess with multiprocessing

Question

Here is my test code to try executing multiple process :

from multiprocessing import Process, Pool, Queue
a = []
def long_time_task(name):
    print 'run task %s (%s)'  % (name,os.getpid())
    start = time.time()
    time.sleep(random.random() * 3)
    end = time.time()
    a.append(name)
    print a

if __name__ == '__main__' :
    print 'parent process %s' % os.getpid()
    p = Pool(5)
    for i in range(10):
        p.apply_async(long_time_task,args = (i,))
    print 'waiting for all subprocess done'
    print a
    p.close()
    p.join()
    print 'all subprocesses done'

From each subprocess, I could get printed different segments at the end after p.join(), like [4, 7],[2],[1, 6],[3, 5],[0, 8, 9]. But the problem is how to aggregate those segments into one entity [0,1,...,10] to print?

Any thoughts would be helpful.

Answer 1

What you are looking at is communication between processes. You can use a queue like it is described in the documentation here:

https://pymotw.com/2/multiprocessing/communication.html

Answer 2

Each process will have its own copy of the a list, so they won't be appending to the same object and your main process won't get the results. You need to use a multiprocessing.Queue and enqueue your results there instead. Then, your main process can dequeue the results and process them as needed.