Running jobs on computer cluster

author

2016-12-05 23:27 (updated 2022-07-22 04:34)

Environment Modules

The Environment Modules package is used to dynamically modify user’s shell environment to support different software applications and different versions of the same software. Common commands include:

module list # list loaded modules
module avail # show available modules
module add OR load # load a module
module rm OR unload # unload a module
module switch OR swap # load another version of a loaded module
module display OR show # display the changes made by a given module
module whatis # brief description of a module
module apropos OR keyword # search for a keyword

More documentation can be found here. Some HPC sites use Lmod, which supports much of the same commands but also adds other functionalities.

Write modulefiles to organize software packages you have installed

You may want to use the Modules system to organize software packages you have installed yourself (e.g., in your home directories). This may be useful if you have installed two versions of the same program or if two packages have executables by the same name. To do this, you need to write a modulefile. You can use module show to find out the location of an existing modulefile and use that as a template to adapt to your own needs.

As an example, on Unity, the parallel_studio_xe/2020.2 modulefile was removed so all the scripts loading this module will fail. However, the underlying software folder actually exists on Unity. Pretending that this is what you have installed, you can simply add a modulefile to use it.

rsync -avzP /home/common/modulefiles/unitymodules/parallel_studio_xe unity:/work/pengbai_umass_edu/unitymodules/
# Other useful modulefiles
rsync -avzP /home/common/modulefiles/privatemodules unity:./

You will also need to add the following line to your ~/.bashrc file on Unity (or to your sub-ABC-unity.script files) so that the Modules system will search the /work/pengbai_umass_edu/unitymodules and ~/privatemodules folders for additional modulefiles.

[ -d '/work/pengbai_umass_edu/unitymodules' ] && module use --append /work/pengbai_umass_edu/unitymodules
# Other useful modulefiles
[ -d "$HOME/privatemodules" ] && module use --append ~/privatemodules

Module that requires additional permissions

Bridges (PSC) and Comet (SDSC) have a “matlab” module for running MATLAB jobs, but it requires additional permissions. For Bridges, fill out this form here. For Comet, follow these instructions to 1) check if you are already in the MATLAB group on Comet; and 2) email help@xsede.org to request to be added to the MATLAB group. In all cases, remember to run MATLAB only on interactive sessions/compute nodes. Do NOT use the login node to open MATLAB or run any scripts.

Job scheduler

PBS/Torque/Maui

Torque/Maui are the PBS systems used for job scheduling. Useful commands:

## Torque Commands ##
qsub # submit a job
qdel # delete a job
qhold # put a job on hold
qalter # modify job attributes
qstat -q # view available queues
qstat # view all queued jobs
qstat -f # view detailed information about a given job

## Maui Commands ##
canceljob # cancel a job
sethold # set job holds
releasehold # release job defers and holds
mjobctl # control and modify job
showq # view all queued jobs
checkjob # check job status
showstart # show estimated start time of a job

Script for a single job

Place the following bash script in the same directory as the input files. If you have prepared fort.4.melt, fort.4.cool, fort.4.volume, fort.4.equil1, this example script can be used to run the melting, cooling, volume equilibration, and full equilibration consecutively in a single job.

#!/bin/bash -l
#PBS -l nodes=1:ppn=1,walltime=24:00:00
# "name" is what will be shown in the queuing system (such as the output from qstat or showq)
#PBS -N name
#PBS -e stderr-$PBS_JOBID
#PBS -o stdout-$PBS_JOBID
###PBS -q devel

# "PROG" points to the executable
PROG=$HOME/path/to/topmon
FILELOG=log
# If you plan to submit this file directly, cd $PBS_O_WORKDIR can be used.
cd "$( dirname "${BASH_SOURCE[0]}" )"

# Load any necessary modules. This is not needed if you have placed it in the ~/.bashrc file.
#module add intel/12.1 ompi/1.4.4/intel mkl/10.2.7.041

# "FCUR" loops over a sequence of strings representing different phases of the simulation.
# If there is a corresponding fort.4.$FCUR file, it will be copied to fort.4 to be used as
# the input file; otherwise the existing fort.4 file will be used.
for FCUR in melt cool volume equil1;do
    echo "$FCUR: " >> $FILELOG
    [ -e "fort.4.$FCUR" ] && cp -f "fort.4.$FCUR" fort.4
    /usr/bin/time -ao $FILELOG $PROG || exit -1
    #/usr/bin/time -ao $FILELOG mpirun -np 4 $PROG || exit -1
    cp -f config1a.dat fort.77
    mv -f run1a.dat "run.$FCUR"
    mv -f config1a.dat "config.$FCUR"
    mv -f movie1a.dat "movie.$FCUR"
    mv -f fort.12 "fort12.$FCUR"
    for j in 1 2 3;do
        mv -f "box${j}config1a.xyz" "box${j}config.$FCUR"
        mv -f "box${j}movie1a.xyz" "box${j}movie.$FCUR"
    done
done

exit 0

Script for multiple jobs (masqueraded as a single one)

To get over the limit that the PBS scheduler may have on the user, the following script can be used. For it to work, the directory should be structured as projectX/temperature1/pressure3/RUN5, etc., where RUN5 contains a file called “sub.script” which is the script in the above section, and projectX contains this example script and is where you should be when submitting the script.

NOTE: There is no need to specify memory (or to use the often incorrectly used singlejob directive) when all cores in a node have been requested. In this case, you will simply get whatever the node has. If the code indeed needs a fair amount of memory to run and the cluster is a heterogeneous one, where a fraction of the nodes satisfy this requirement while others don’t, memory can be requested using pmem=X on a per core basis or mem=X for the entire submission script. Note this will prevent the job being allocated to the compute nodes that do not have enough memory.

#!/bin/bash -l
#PBS -l nodes=6:ppn=8,walltime=24:00:00
#PBS -N name
#PBS -e stderr-name
#PBS -o stdout-name
###PBS -q devel

DATA=$PBS_O_WORKDIR
cd $DATA

NODE=($(cat $PBS_NODEFILE | uniq))
#echo ${NODE[@]} > tmp-$PBS_JOBID

inode=-1
ijob=0

for I in temperature1 temperature2; do
    for J in pressure1 pressure2 pressure3; do
# assuming 8 independent runs
        for ((K=1;K<=8;K++)); do
# How many independent jobs can one node run? Suppose a node has 8 cores, then if your
# simulations use 1 core/job, the number is 8, or if your jobs use 4 cores/job, the
# number is 2, and so on.
# Option 1
            [ $((ijob++ % 8)) -eq 0 ] && ((inode++))
            ssh ${NODE[inode]} $DATA/$I/$J/RUN$K/sub.script &
# or (replacing only the previous line)
#            pdsh -w ${NODE[inode]} $DATA/$I/$J/RUN$K/sub.script &
# Option 2 (replacing lines above up to "Option 1", and the inode variable no longer needed)
#            pbsdsh -n $((ijob++)) $DATA/$I/$J/RUN$K/sub.script &
        done
    done
done

wait

exit 0

NOTE: Certain ssh implementations have a hard time with very many ssh connections issued in a short period of time, resulting in jobs not run and errors like “Connection closed by …”. To get around this problem, place the lines below to the ~/.ssh/config file.

# Limit these settings to Itasca
Host node*
  ConnectionAttempts 8000
# The following alternative uses multiplexing
#   ControlMaster auto
#   ControlPath ~/mux-%l-%r@%h:%p
#   ControlPersist 30m

Improving performance of Lustre file system

Lustre stripes files across object storage targets to achieve good parallel performance and high bandwidth. However, if the application only uses one process for I/O, then the benefits of striping will not be realized and network latency and congestion will usually lead to worse I/O performance. For these applications, it is best to limit stripe count to 1:

find /lustre/user_name/ -type d -exec lfs setstripe -c 1 {} \; # only affects files and directories newly created

and enable I/O caching:

export FORT_BUFFERED=1

The above command should be put in all job submission scripts or in the ~/.bashrc file.

Heterogeneous cluster

The group cluster is a Beowulf cluster of heterogeneous resources built over the years. The operating system is Rocks Cluster Linux distribution.

Note that programs compiled on the new E5-2609 CPUs will not run on the old E54xx series and the frontend is an old machine. If you need to submit a large number of jobs that will spread across different types of nodes, then you need to ssh to a node of each type (The new nodes are named compute-0-**, and the old nodes are named compute-x-**, x>0), compile your code, and name it differently, such as topmon and topmon-E54xx. Then in your submission script, you can put something like this:

PROG=$HOME/MCCCS/exe/topmon
[[ ! $(hostname) =~ compute-0-* ]] && PROG=${PROG}-E54xx