Preallocating memory for variable that changes size every iteration (long text file)

Question

I wrote a code which translates an image to text file for lithography. The images are high resolution (26.5k x 26.5k) and the text files are over 2MB. My problem is that it takes very long for the code to work and I suspect it has something to do with memory fragmentation (I am not an experienced programmer and specifically just started working with MATLAB). It lags in two places: before you have to pick the specific image you want to use, and after when it processes it.

In the processing part I get this comment from MATLAB:

The size of the indicated variable or array appears to be changing with each loop iteration. Commonly, this message appears because an array is growing by assignment or concatenation....

But then there is also this part:

If any of the following conditions are true, it might be appropriate to suppress this message, as described in Adjust Code Analyzer Message Indicators and Messages: The loop code contains a conditional block where the array grows. (In this case, it can be reasonable to grow the array only as the loop finds such conditions.) For each iteration in the loop, the length of the concatenation varies (such as character vectors with variable length). The total size of the array is not calculable before entering the loop.

And the output text file depends a lot on the image provided and number of black/white pixels in it.

So I wonder if there is any way to improve the performance of the code?

Before choosing the image file, there are almost no calculations at all, and the ones which are carried are very simple, so I don't know why it's so slow.

For the first function, just before calling the second one which opens the image file:

function print_contacts(pixl, size, xRd, yRd, zRd, xRu, yRu, zRu)

meas_L=3960.05; % the lateral and veritcal measured distance e.g. 4900.5 between large pluses, 3960.05 between dots in corners
%%
conv=(size*1000)/pixl;%0.267;%0.4215; % pixel to microns
siz=pixl*conv;
xLd=0.0;
yLd=0.0;
zLd=0.0;

%%
delta_x_H=xLd-xRd;
delta_y_H=yLd-yRd;
delta_z_V=-(zLd-zRd);

delta_x_V=xRu-xRd;
delta_y_V=yRu-yRd;
delta_z_H=-(zRu-zRd);

%% find x-y angle of rotation teta (if teta is positive then rotation is clockwise)

if yRd > 0
    teta_xy=atand(-xRd/yRd)-90;
else
    teta_xy=atand(-xRd/yRd)+90;
end

%% find x-z angle of rotation teta (if teta is positive then rotation is clockwise)

teta_xz=atand(delta_z_H/delta_x_H);

%% find y-z angle of rotation teta (if teta is positive then rotation is clockwise)

teta_yz=atand(delta_z_V/delta_y_V);

%%
del_z_H=delta_z_H/meas_L*conv*pixl;
del_z_V=delta_z_V/meas_L*conv*pixl;

vec_xz=linspace(0,del_z_H,pixl);
vec_yz=linspace(0,del_z_V,pixl);
[X,Y] = meshgrid(vec_yz,vec_xz);
Z1=(X + Y);
shift=Z1(pixl/2,pixl/2);
valZ=round((Z1-shift),2);
%surf(valZ);
con=[conv];
R = [cosd(teta_xy) -sind(teta_xy); sind(teta_xy) cosd(teta_xy)];

PPM.Y=con;
PPM.X=con;

regionpropsApply(PPM, valZ, siz, R, conv)

After which comes the function regionpropsApply which lets you choose the image file and fragments it into individual images each of which is going to be processed into a text file by the 3rd and last function, in which I have this command which collects a long string which will go into the text file:

str2File = [str2File;cmdJump;cmdZ;cmdMov];

Where cmdJump, cmdZ and cmdMov are the commands created in this specific iteration. Before iterating, I already know the number of required iterations. Each iteration will include these 3 commands always. Each image (fragmented from the original one) is processed separately. When should I preallocate memory and how?

Currently, executing the code takes over 40 minutes, and considering the fact the output file is only 2MB, I really wish it would take less than a few minutes.

Answer 1

In your first bit of code, I presume that this is the slow part:

vec_xz=linspace(0,del_z_H,pixl);
vec_yz=linspace(0,del_z_V,pixl);
[X,Y] = meshgrid(vec_yz,vec_xz);
Z1=(X + Y);
shift=Z1(pixl/2,pixl/2);
valZ=round((Z1-shift),2);

Since you are dealing with very large images, I presume pixl here is a large number. You generate 4 matrices of the size of your image. But 3 of these contain intermediate data, which is used only within the code above. Instead of assigning each computation result to a new variable, re-use variables to avoid using too much memory. For example, calling a function with x = func(x) allows the function to work in-place, modifying x instead of copying it. Now you should no longer have Z1 around taking up memory.

The images X and Y are completely superfluous. Since R2016b, you can simply add two orthogonal vectors, and for 10 years before that we had bsxfun for that type of operation. This not only saves memory, it also speeds up computation by better using the cache.

Thus, you can do the same as you did above by:

vec_xz = linspace(0, del_z_H, pixl);
vec_yz = linspace(0, del_z_V, pixl);
valZ = vec_xz + vec_yz.';
shift = valZ(pixl/2, pixl/2);
valZ = round(valZ-shift, 2);

If you have an older version of MATLAB then replace the 3rd line with:

valZ = bsxfun(@plus, vec_xz, vec_yz.');

Finally, we can reduce the number of operations by changing their order: instead of subtracting shift from each pixel, we subtract a part from each of the two vectors:

vec_xz = linspace(0, del_z_H, pixl);
vec_xz = vec_xz - vec_xz(pixl/2);
vec_yz = linspace(0, del_z_V, pixl);
vec_yz = vec_yz - vec_yz(pixl/2);
valZ = vec_xz + vec_yz.';
valZ = round(valZ, 2);

---

As for the second part of your question, I don’t know what else goes on there and what you do with the string you build, but if all you do is build this long string and then write it to file, you should instead write it to file as you go:

file = fopen('name.txt', 'wt');
for ...
   fwrite(file, [cmdJump;cmdZ;cmdMov];
end
fclose(file);

If you need to modify the strings some way before writing, you could store them in a cell array, which you can preallocate because you know the number of iterations of your loop:

str2File = cell(N,1);
for ii=1:N
   str2File{ii} = [cmdJump;cmdZ;cmdMov];
end