I'm running a function in parallel with python multiprocessing module. This function takes images as input, and outputs some measurements. The problem is that, I compared the results of time.time() vs time.clock(), it seems the wall-time is WAY longer than the cpu-time. Something like 240 seconds vs. 0.22 seconds!!
If I understand correctly, this means CPU spends less than 1 second on executing my function, and for most of the time, I'm waiting for reading/writing/disk etc. Does this sound normal? I'm using a public super-computing center (it's very powerful though), so I'm not sure whether it was because other users are also using the cpu nodes and I had to wait?
The following is my code, could you point out if any part of the code might be the cause? Thank you very much!
import cv2
import glob
from joblib import Parallel,delayed # this is for parallel computing
import time as t
from Function_ImageMeasure_sz import salinency, color_HSV,rot_balance
mypath='C:/Research/Data/Images/*.jpg'
# define function that will be executed in parallel
def get_measurement(file_name):
img = cv2.imread(file_name) # read image
# the following steps are computing some metrics over the input image
contours,ind,centroid=salinency(img)
d_rot,balance=rot_balance(img,centroid)
color_rank, color_tone,bright_stat,saturate_stat,clear_dull,symmetry=color_HSV(img)
result=[d_rot,balance,centroid,color_rank,color_tone,bright_stat,saturate_stat,clear_dull,symmetry]
return result
# call function
files=glob.glob(mypath) # input all images in the same folder
njob=2 # define how many threads/pipelines in parallel
t1=t.clock()
t3=t.time()
# this is implement the function in parallel
results=Parallel(n_jobs=njob,backend='multiprocessing')(map(delayed(get_measurement),files))
t2=t.clock()
t4=t.time()
# output results
print 'processing %s image takes %s seconds (wall-time), %s seconds (cpu-time) with %s threads/cores'%(len(files),t4-t3,t2-t1,njob)`