How can I output my SBATCH options in the .out file OR How to echo commented lines in bash

Question

I'm fairly new to HPC computing using the SLURM Workload Manager and hence have yet to find a reliable working routine. As of now, I have a jobscript with a bunch of #SBATCH flags that I modify whenever I do another calculation. This includes the total run time and the number of nodes / tasks / CPUs.

Since I'm still gaining experience, I would like to save the chosen options for reference in the standard .out - file produced by SLURM. I have found on the sbatch manual page a few options in the filename pattern section to at least remember the jobname, but what I really want is a complete list of all the set #SBATCH flags.

My current workaround goes like this

#!/bin/bash
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=4
#SBATCH --time=0:20:00
#SBATCH --mem=4G
#SBATCH -p normal
#SBATCH -q cont
#SBATCH -J crosscorr
#SBATCH --output=jobscript_slurmout/R-%j.%x.out

echo "#SBATCH --ntasks=2
#SBATCH --cpus-per-task=4
#SBATCH --time=0:20:00
#SBATCH --mem=4G
#SBATCH -p normal
#SBATCH -q cont
#SBATCH -J crosscorr
#SBATCH --output=jobscript_slurmout/R-%j.%x.out
"

and is obviously not satisfying and also prone to typos. Since the SLURM manager relies on code that bash does not see (as it's commented out), I see no easy fix using the bash language for this. Neither did I find any hints using the sbatch options.

Maybe there even is another way to save those options. I know that I might grep these desired options using a automatized seff inquiry, but I wish there would be an easier way, because the Job ID is only assigned after job submission.

Answer 1

I don't know what slurm is but you appear to be using a construct #SBATCH to set some options for it within what is otherwise a bash script so maybe you could populate a bash array with the options and then both print them and #SBATCH them, e.g. (the spaces at the start of the here-document lines being read by readarray must be tabs, not blanks):

$ cat tst.sh
#!/usr/bin/env bash

readarray -t opts <<-'!'
        --ntasks=2
        --cpus-per-task=4
        -time=0:20: 00
        --mem=4G
        -p normal
        -q cont
        -J crosscorr
        --output=jobscript_slurmout/R-%j.%x.out
!

for opt in "${opts[@]}"; do
    printf '#SBATCH %s\n' "$opt"
    #SBATCH "$opt"
done

$ ./tst.sh
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=4
#SBATCH --time=0:20:00
#SBATCH --mem=4G
#SBATCH -p normal
#SBATCH -q cont
#SBATCH -J crosscorr
#SBATCH --output=jobscript_slurmout/R-%j.%x.out

You can use that to either replace your current script to set #SBATCH if that works or to generate another script to replace your current script:

$ cat tst.sh
#!/usr/bin/env bash

readarray -t opts <<-'!'
        --ntasks=2
        --cpus-per-task=4
        -time=0:20: 00
        --mem=4G
        -p normal
        -q cont
        -J crosscorr
        --output=jobscript_slurmout/R-%j.%x.out
!

printf '#!/usr/bin/env bash\n\n'
printf '#SBATCH %s\n' "${opts[@]}" | tee '/dev/stderr'

$ cat slurm.sh
cat: slurm.sh: No such file or directory

$ ./tst.sh > slurm.sh
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=4
#SBATCH --time=0:20:00
#SBATCH --mem=4G
#SBATCH -p normal
#SBATCH -q cont
#SBATCH -J crosscorr
#SBATCH --output=jobscript_slurmout/R-%j.%x.out

$ cat slurm.sh
#!/usr/bin/env bash

#SBATCH --ntasks=2
#SBATCH --cpus-per-task=4
#SBATCH --time=0:20:00
#SBATCH --mem=4G
#SBATCH -p normal
#SBATCH -q cont
#SBATCH -J crosscorr
#SBATCH --output=jobscript_slurmout/R-%j.%x.out

or it looks like your file of #SBATCH directives is used by a tool named opts so maybe you could do:

#!/usr/bin/env bash

readarray -t opts <<-'!'
        --ntasks=2
        --cpus-per-task=4
        -time=0:20: 00
        --mem=4G
        -p normal
        -q cont
        -J crosscorr
        --output=jobscript_slurmout/R-%j.%x.out
!

sbatch <(
    printf '#!/usr/bin/env bash\n\n'
    printf '#SBATCH %s\n' "${opts[@]}" | tee '/dev/stderr'
)

to use command substitution to provide a "file" of #SBATCH directives to sbatch, or if sbatch requires a regular file:

$ cat ./tst.sh
#!/usr/bin/env bash

tmp=$(mktemp) || exit
trap 'rm -f "$tmp"; exit' EXIT

readarray -t opts <<-'!'
        --ntasks=2
        --cpus-per-task=4
        -time=0:20: 00
        --mem=4G
        -p normal
        -q cont
        -J crosscorr
        --output=jobscript_slurmout/R-%j.%x.out
!

printf '#!/usr/bin/env bash\n\n'
printf '#SBATCH %s\n' "${opts[@]}" | tee "$tmp" >&2

sbatch "$tmp"

to create the file of #SBATCH commands, call sbatch with that file as an argument, then remove that file.

all the the | tee ...s above are just so you can see the #SBATCH directives being generated as I think that's mainly what you wanted.