Examples of sbatch commands

We have now completely transitioned to the Slurm scheduler, running on fortyfour.ibest.uidaho.edu (and fortyfive.ibest.uidaho.edu). Slurm is better equiped to deal with our now more heterogenious node environment, and provides additional fault tolerance. The old head node, fortythree.ibest.uidaho.edu, is no longer available.

Example scripts are provided for both slurm and the older PBS/torque to assist users in transitioning from PBS/Torque to slurm. For more of a walkthrough on cluster usage - see this workshop documentation.

A submission script that echoes the obligatory ‘Hello World’

Works with both PBS and slurm

 #!/bin/bash
 echo "Hello World! From $HOSTNAME"

Output

A submission script that sleeps for 1 second 30 times while printing out the count to both stderr and stdout.

By default, Slurm combines the standard output and standard error streams, so for this example, we need to split them.

#!/bin/bash
#SBATCH -o stdout.out
#SBATCH -e stderr.out
for i in `seq 1 30` ; do
     echo "Hello from stdout on $HOSTNAME: $i"
     echo "Hello from stderr on $HOSTNAME: $i" 1>&2
     sleep 1
done

PBS

 #!/bin/bash
 for i in `seq 1 30` ; do
      echo "Hello from stdout on $HOSTNAME: $i"
      echo "Hello from stderr on $HOSTNAME: $i" 1>&2
      sleep 1
 done

Requesting resources

We have a heterogeneous node environment, with several different types of nodes:

Nodes Count Cores RAM Other
n001-3,n010-15,n020-7,n031-37,n054-60~30 8 32G
n005-9 5 24 80G
n018-19,n038,n053 4 16 96G
n004,n016 2 32 192G
n028-030 3 48 512G
n039-052 14 40 256G
n061 1 4 48G
n063,n064 2 16 48G
n065-066 2 40 32G 4 NVIDIA Tesla K80 GPUs, IMCI use only
n067-069 2 48 192G
n070-101 32 16 128G
n102,n103 2 32 128G
n104 1 8 64G 1 NVIDIA 1080Ti GPU
n105,110-113 6 16 128G 2 NVIDIA 1080Ti GPU
n106 1 40 64G 2 NVIDIA T4
n114 1 16 128G 2 NVIDIA RTX 2080Ti GPU
n115-116 2 40 192G 2 NVIDIA Tesla V100 GPUs, IMCI use only
n117 1 16 128G
n118 1 24 128G 1 NVIDIA Titan RTX GPU
n119 1 40 192G
n120 1 64 384G
n121-122 2 64 192G 2x Nvidia T4
n123 1 128 256G 4x Nvidia A100, IMCI use only
n124 1 40 192G 2x Nvidia A100, gpu-volatile

By default, jobs will run on nodes 070-103 if they are free, (as long as those nodes meet the job requirements). In order that slurm doesn’t crash nodes by oversubcribing their available RAM, each job is allocated 3GB of RAM by default. If your job exceeds this amount, slurm will forcibly stop it. Use the –mem-per-cpu= or –mem= arguments to request more. (Note: the –mem-per-cpu argument is in MB, whereas the –mem argument will accept suffixes M | G | T). If your job requires more than 128 GB of RAM, you can request more, and your job will run on nodes that have more RAM.

benji@fortyfour ~ $ sbatch --mem=150G my_job_script.slurm 
Submitted batch job 727
benji@fortyfour ~ $ cat slurm-727.out 
running on 
n067
 [1] -0.607378506 -0.098118967  1.346261686 -0.035944115  0.017377095
 [6]  0.158619002  0.971589403  0.004847043 -0.190260312 -0.226918163
finished
benji@fortyfour ~ $

If you want your job to use more than one cpu thread (core), you should explicitly request more cores so that nodes do not end up overloaded.

benji@fortyfour ~ $ sbatch --cpus-per-task=32 --mem-per-cpu=5000 my_job_script.slurm 
Submitted batch job 731
benji@fortyfour ~ $ cat slurm-731.out 
running on 
n067
 [1]  0.077898249 -1.111838945  0.327516862 -0.358068017  1.810896784
 [6] -0.071178293  0.012828645 -1.484885936  0.637533051  0.004092041
finished
benji@fortyfour ~ $ sbatch --cpus-per-task=16 my_job_script.slurm 
Submitted batch job 732
benji@fortyfour ~ $ cat slurm-732.out 
running on 
n090
 [1] -1.4791174  0.2250128  0.3104545  0.3461544  0.9653486  0.4038346
 [7] -0.5698751  1.1166059 -0.6974919 -0.2849535
finished
benji@fortyfour ~ $

These options can be specified in the submission script as well:

#!/bin/bash

#SBATCH --mem=150G
#SBATCH --cpus-per-task=16

cd $SLURM_SUBMIT_DIR

echo "running on "
hostname

source /usr/modules/init/bash
module load R

Rscript -e "rnorm(10)"

echo "finished"

To request nodes that mount Ceph, use the -C “ceph” option:

benji@fortyfour ~ $ sbatch -C "ceph" my_job_script.slurm

Job Name

Use the -J flag

#!/bin/bash
#SBATCH -J sleep
#SBATCH -o stdout.out
#SBATCH -e stderr.out

for i in `seq 1 30` ; do
     echo "Hello from stdout on $HOSTNAME: $i"
     echo "Hello from stderr on $HOSTNAME: $i" 1>&2
     sleep 1
done

PBS

PBS uses the -N flag

#!/bin/bash
#PBS -N sleep

for i in `seq 1 30` ; do
     echo "Hello from stdout on $HOSTNAME: $i"
     echo "Hello from stderr on $HOSTNAME: $i" 1>&2
     sleep 1
done

Modify the same submission script to ‘join’ the output files into one file called myoutput.log

#!/bin/bash
#SBATCH -J sleep
#SBATCH -o myoutput.log

for i in `seq 1 30` ; do
     echo "Hello from stdout on $HOSTNAME: $i"
     echo "Hello from stderr on $HOSTNAME: $i" 1>&2
     sleep 1
done

Take the last submission script and tell qsub/sbatch to execute it 5 minutes from now.

sbatch commands

$ sbatch --begin=now+5minutes sleep3.slurm

PBS

#!/bin/bash
#PBS -N sleep
#PBS -j oe
#PBS -o myoutput.log

for i in `seq 1 30` ; do
     echo "Hello from stdout on $HOSTNAME: $i"
     echo "Hello from stderr on $HOSTNAME: $i" 1>&2
     sleep 1
done

qsub commands

$ FUTUREDATE=`date -d "5 minutes" +%Y%m%d%H%M`
$ qsub -d ./ -a $FUTUREDATE sleep3.pbs

Email Job Status

Modify sleep3.pbs to mail you when the cluster begins, ends, and aborts execution.

#!/bin/bash
#SBATCH -J sleep
#SBATCH -o myoutput.log
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH --mail-user=rlyon@uidaho.edu

for i in `seq 1 30` ; do
     echo "Hello from stdout on $HOSTNAME: $i"
     echo "Hello from stderr on $HOSTNAME: $i" 1>&2
     sleep 1
done

PBS

#!/bin/bash
#PBS -N sleep
#PBS -j oe
#PBS -o myoutput.log
#PBS -m abe
#PBS -M rlyon@uidaho.edu

for i in `seq 1 30` ; do
        echo "Hello from stdout on $HOSTNAME: $i"
        echo "Hello from stderr on $HOSTNAME: $i" 1>&2
        sleep 1
done

Slurm Partitions (Queues)

Instead of ‘queues’, the Slurm scheduler (fortyfour) has ‘partitions’. There are now a few partitions available. The reg partition is the default. The gpu-short partition has a single GPU node, and is intended for testing/development of your scripts/jobs. The gpu-long partition has a time limit of 1 week. Please do not submit more than 10 or so jobs to the gpu-long queue so that other users can have a chance at using the resource as well.

PartitionWall-timeMax-JobsNodes
tiny 6 hours 10000 40
short 24 hours 1000 40
reg 168 hours 500 100
long infinite 50 100
gpu-short 24 hours 4 1
gpu-long 168 hours 6 8
volatile 128 hours 50 19

Sometimes the partition configuration changes quicker than this documentation, but you can run the sinfo -s command and get the current configuration:

benji@fortyfour ~ * sinfo -s
PARTITION   AVAIL  TIMELIMIT   NODES(A/I/O/T)  NODELIST
primary     inact   infinite          0/3/0/3  n[091-093]
tiny           up    6:00:00       20/19/1/40  n[067-103,107-109]
short          up 1-00:00:00       20/19/1/40  n[067-103,107-109]
testded        up   infinite          0/2/0/2  n[018,062]
seven       inact   infinite       20/78/1/99  n[001-017,019-061,063-064,067-090,094-103,107-109]
reg*           up 7-00:00:00      20/79/1/100  n[001-017,019-061,063-064,067-103,107]
long           up   infinite      20/79/1/100  n[001-017,019-061,063-064,067-103,107]
gpu-short      up 1-00:00:00          0/1/0/1  n104
gpu-long       up 7-00:00:00          1/4/2/7  n[105-106,110-114]
submit-only    up   infinite          0/0/4/4  acheron,jayne,oceanus,styx
benji@fortyfour ~ *

To select a partition, use the -p command line argument.

#!/bin/bash
#SBATCH -J sleep
#SBATCH -o myoutput.log
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH --mail-user=rlyon@uidaho.edu
#SBATCH -p primary

for i in `seq 1 30` ; do
     echo "Hello from stdout on $HOSTNAME: $i"
     echo "Hello from stderr on $HOSTNAME: $i" 1>&2
     sleep 1
done

Environment Variables

Simple submission scripts to print out all the ‘extra’ environment variables that Slurm provides.

This is not an exhaustive list of the available environment variables. For the full list, consult `man sbatch’

#!/bin/bash
#SBATCH -J vars

echo "SLURM_SUBMIT_HOST: $SLURM_SUBMIT_HOST"
echo "SLURM_SUBMIT_DIR: $SLURM_SUBMIT_DIR"
echo "SLURM_ARRAY_TASK_ID: $SLURM_ARRAY_TASK_ID"
echo "SLURM_JOBID: $SLURM_JOBID"
echo "SLURM_JOB_NAME: $SLURM_JOB_NAME"
echo "SLURM_JOB_NODELIST: $SLURM_JOB_NODELIST"
echo "SLURM_JOB_PARTITION: $SLURM_JOB_PARTITION"

A submission script that prints out the environment using the ‘env’ command.

#bin/sh
env

With sbatch, look at the differences between submitting normally, with --get-user-env, with --export=NONE, and with --export=FOO=bar. Note that by default, Slurm exports all of your environment variables to the job. This is different from Torque’s behavior.

PBS

#!/bin/bash
#PBS -N vars

echo "PBS_O_HOST: $PBS_O_HOST"
echo "PBS_SERVER: $PBS_SERVER"
echo "PBS_O_QUEUE: $PBS_O_QUEUE"
echo "PBS_O_ROOTDIR: $PBS_O_ROOTDIR"
echo "PBS_O_WORKDIR: $PBS_O_WORKDIR"
echo "PBS_ARRAYID: $PBS_ARRAYID"
echo "PBS_ENVIRONMENT: $PBS_ENVIRONMENT"
echo "PBS_JOBID: $PBS_JOBID"
echo "PBS_JOBNAME: $PBS_JOBNAME"
echo "PBS_NODEFILE: $PBS_NODEFILE"
echo "PBS_QUEUE: $PBS_QUEUE"

A submission script that prints out the environment using the ‘env’ command.

#bin/sh
env

With qsub, look at the differences between submitting normally, with -V, and with -v FOO=bar.


Load Modules

A script to load the mrbayes module and run mb data/primates.nex

#!/bin/bash
#SBATCH -J mrbayes

. /usr/modules/init/bash
module load mrbayes

cd $SLURM_SUBMIT_DIR

mb data/primates.nex

Modify the mrbayes script created previously to accept a variable called FILE which supplies the nex file to the script.

#!/bin/bash
#SBATCH -J mrbayes

. /usr/modules/init/bash
module load mrbayes

cd $SLURM_SUBMIT_DIR

mb $FILE

To submit:

sbatch --export=FILE=data/primates.nex mb-gen.slurm

PBS

The script

#!/bin/bash
#PBS -N mrbayes

. /usr/modules/init/bash
module load mrbayes

cd $PBS_O_WORKDIR

mb data/primates.nex

Modify the mrbayes script created previously to accept a variable called FILE which supplies the nex file to the script.

#!/bin/bash
#PBS -N mrbayes

. /usr/modules/init/bash
module load mrbayes

cd $PBS_O_WORKDIR

mb $FILE

To submit:

qsub -d ./ -v FILE=data/primates.nex

Array jobs

Submission scripts to convert the ‘ffn’ files in ‘fasta’ to embl using an array job.

indexdir.sh

#!/bin/bash
SRC=$1
DST=$2
SUFFIX=$3
COUNT=1

for FILE in $SRC/*.${SUFFIX} ; do
        ln -sf "$PWD/$FILE" "$DST/${COUNT}.seq"
        COUNT=$((COUNT + 1))
done

array1.slurm

#!/bin/bash
#SBATCH -J convert
#SBATCH -o logs/convert

cd $SLURM_SUBMIT_DIR
# Use variables to get source and destination
if [ ! -d $SRC ] ; then
        echo "Source directory does not exist"
        exit 1
fi

if [ ! -d $DST ] ; then
        echo "Destination directory does not exist"
        exit 1
fi

. /usr/modules/init/bash
module load emboss

LINKEDNAME=$(basename `readlink $SRC/$SLURM_ARRAY_TASK_ID.seq` | cut -d'.' -f1)
seqret $SRC/$SLURM_ARRAY_TASK_ID.seq $DST/$LINKEDNAME.seq -osf $FORMAT

To submit:

sbatch --export=SRC=indir,DST=outdir -a 1-10 array1.slurm

A submission script that runs mb on all the nex files in ‘data’ using an array job

You can reuse the indexing code from the previous example.

array2.slurm

#!/bin/bash
#SBATCH -J mb-array
#SBATCH -o logs/mb-array

cd $SLURM_SUBMIT_DIR
# Use variables to get source and destination
if [ ! -d $SRC ] ; then
        echo "Source directory does not exist"
        exit 1
fi

. /usr/modules/init/bash
module load mrbayes

mb ${SRC}/${SLURM_ARRAY_TASK_ID}.nex

To submit:

sbatch --export=SRC=indir -a 1-5 mb-array.slurm

PBS

array1.pbs

#!/bin/bash
#PBS -N convert
#PBS -j oe
#PBS -o logs/convert

cd $PBS_O_WORKDIR
# Use variables to get source and destination
if [ ! -d $SRC ] ; then
        echo "Source directory does not exist"
        exit 1
fi

if [ ! -d $DST ] ; then
        echo "Destination directory does not exist"
        exit 1
fi

. /usr/modules/init/bash
module load emboss

LINKEDNAME=$(basename `readlink $SRC/$PBS_ARRAYID.seq` | cut -d'.' -f1)
seqret $SRC/$PBS_ARRAYID.seq $DST/$LINKEDNAME.seq -osf $FORMAT

To submit:

qsub -d ./ -v SRC=indir,DST=outdir -t 1-10 array1.pbs

mb-array.pbs

#!/bin/bash
#PBS -N mb-array
#PBS -j oe
#PBS -o logs/mb-array

cd $PBS_O_WORKDIR
# Use variables to get source and destination
if [ ! -d $SRC ] ; then
        echo "Source directory does not exist"
        exit 1
fi

. /usr/modules/init/bash
module load mrbayes

mb ${SRC}/${PBS_ARRAYID}.nex

To submit:

qsub -d ./ -v SRC=indir -t 1-5 mb-array.pbs