Wildly different iteration times in parfor loop

Question

I have an embarrassingly-parallel Monte Carlo code for which I am using parfor. Each parfor loop iteration does around 100 seconds of calculation into a temporary array, before adding that array to the master array, which is an reduction variable. I'm timing using a sliced variable.

Like this:

master_array=zeros(1000,1000,50)
iteration_times=zeros(500,1);

parfor i=1:500
    iteration_start=datetime;

    temp_array=zeros(1000,1000,50);

    **do about 100 seconds of work building temp_array**

    master_array=master_array+temp_array;

    iteration_times(i)=seconds(datetime-iteration_start);

end

Sometimes when the code is run, but not all the time, the code takes far longer. When it does, it seems that some loop iterations are taking way way longer than they should, as shown in the graph iteration number vs runtime. (This is not caused by variations in the amount of 'work done' - the variation disappears without the parfor.) This was running with a pool of 4 workers.

The arrays are large, and can be 100MB or more. I don't seem to be running out of memory - the hard drive is not being used, and I get this problem still with smaller arrays.

'Slow' iteration times seem oddly quantised, like some loop interations are waiting for others to complete before completing themselves.

Any ideas what i can check?