How to submit a job with sbatch (SLURM) when number of ranks per node varies between nodes?

Question

I'm trying to submit a job to a SLURM cluster through sbatch where the number of ranks per node varies between nodes. For example, the first 24 nodes require 4 ranks per node, and the last 12 nodes require 1 rank per node.

How would I go about this?

Answer 1

You can use heterogeneous jobs. In simplified version, your submission script would look like this

#!/bin/bash
#SBATCH --nodes=24
#SBATCH --ntasks-per-node=4
#SBATCH ...
#SBATCH hetjob
#SBATCH --nodes=12
#SBATCH --ntasks-per-node=1
#SBATCH ...
...