Slurm: create a directory on all nodes

Question

I'm launching a job for parallel execution with slurm. This job needs a certain directory structure to exist in each node, but if I use mkdir in the job script, the directories are only created in the first node. How can I make sure the directories are created in all nodes used by the job?

Answer 1

I guess I have to answer myself. This is not a perfect solution, but it worked in my case good enough. In the job script I used this before the real job starts:

for node in $(scontrol show hostnames $SLURM_NODELIST) ; do
  srun -N 1-1 -n 1 -w $node mkdir -p /directory/to/be/created
done
sleep 60

The node list in $SLURM_NODELIST is abbreviated, with the scontrol statement I get the full names. Without the sleep command I sometimes had problems with some directory not existing, so added it just to be safe.

The problem is that I need to know which directories need to be created in advance, which is possible in my case, but might be more difficult in other circumstances.

Answer 2

You can simply add

srun mkdir -p /directory/to/be/created

in your script. It might try and create the same directory several times depending on the cluster configuration, but it will work anyhow.