How to append multiple .mat files that have the same variable names in them?

Question

I run my scripts which produce hundreds or thousands of .mat files. Each of this file contains two variables: resultsU and resultsT. I want to append the files but not overwrite the variables. What is the easiest way to do so in Matlab? Some suggest manual manipulation of the .mat file which is not easy or efficient to do when you have hundreds of .mat files.

Answer 1

rayryeng's answer is good if running the code which created the files is an option. However, if working with a huge amount of files is a given fact which you need to deal with, I would suggest to go about this using and array of structs (which resembles struct concatenation).

Consider the following example function:

function combined_results = CombineMat(newFolder)

oldFolder = cd; %// Backup the current directory
cd(newFolder);  %// Switch to some new folder
fList = dir('*.mat'); fList = {fList.name}'; %'// Get the file list

%% // Processing the list of files:
if isempty(fList), combined_results = []; return, end %// Check that some files exist

%// Initialize the result struct by loading the last file (also preallocates the struct):
nFiles = size(fList,1);
combined_results(nFiles) = load(fullfile(newFolder,fList{1}));

%// See if there is only 1 file, and return if so:
if nFiles == 1, return, end

%// Process any additional files 
for ind1 = 1:nFiles-1
    combined_results(ind1) = load(fullfile(newFolder,fList{ind1}));
end

%% Cleanup - changing the current directory back:
cd(oldFolder);

What it does is to combine .mat files containing the same variable names. Now you can run combined_results = CombineMat(folder_with_mat_files) and get a struct which contains all the different results (assuming you have enough memory to hold them). After you have this struct in memory, you can save it to a single .mat file.

Note 1: If you don't have enough memory to load all the files, you can add another piece of code to CombineMat that dumps combined_results to disk and clears it after a certain amount of loop iterations (possibly with the '-append' option as suggested by rayryeng. This personally doesn't make sense to me if OOM happens, because then you would have problems loading the resulting file :)

Note 2: Obviously, if you want your result as something other than an array of structs, the code needs to be modified accordingly.

---

In a more general case, where you are trying to concatenate structures with different variable names, you can use the following FEX submission (which I can recommend from personal experience!).

---

P.S. My code was written on MATLAB 2015a.

Answer 2

It's actually a lot easier than you think. If you want to append to MAT files, simply use save with the -append flag. Assuming that you have a few variables... let's call them p and q, and assuming you have a file called test.mat, it's very simply:

save('test.mat','p','q','-append');

The beauty of this is that you don't need to load any of the variables in the MAT file and resave them with the appended variables. This appends the desired variables into the MAT file without you ever having to load them into MATLAB.

If you have a bunch of .mat files in a directory, you can do something like this:

folder = '...'; %// Place directory here
f = dir(folder); %// Find files

%// For each file...
for idx = 1 : numel(f)
    name = fullfile(folder, f(idx).name); %// Get path to file

    %// Do some processing
    %//...
    %//

    %// Append to file
    save(name, ..., ..., ..., ..., '-append');
end

What goes inside the ... for save are the variables you want to append to each file.