I am trying to build multiprocessing in python to reduce computation speed, but it seems like after multiprocessing, the overall speed of computation decreased significantly. I have created 4 different processes and split dataFrame into 4 different dataframe, which will be an input to each processes. After timing each process, it seems like the overhead cost is significant, and was wondering if there is way to reduce these overhead costs.
I am using windows7, python 3.5 and my machine has 8 cores.
def doSomething(args, dataPassed,):
processing data, and calculating outputs
def parallelize_dataframe(df, nestedApply):
df_split = np.array_split(df, 4)
pool = multiprocessing.Pool(4)
df = pool.map(nestedApply, df_split)
print ('finished with Simulation')
time = float((dt.datetime.now() - startTime).total_seconds())
pool.close()
pool.join()
def nestedApply(df):
func2 = partial(doSomething, args=())
res = df.apply(func2, axis=1)
res = [output Tables]
return res
if __name__ == '__main__':
data = pd.read_sql_query(query, conn)
parallelize_dataframe(data, nestedApply)
I would suggest to use queues instead of providing your DataFrame as chunks. You need a lot of ressources to copy each chunk and it takes quite some time to do so. You could run out of memory if your DataFrame is really big. Using queues you could benefit from fast iterators in pandas. Here is my approach. The overhead reduces with the complexity of your workers. Unfortunately, my workers are far to simple to really show that, but
sleep
simulates complexity a bit.Using
numProc = 2
it takes 50sec per loop, withnumProc = 4
it is twice as fast.