Shotgun job management system

author

2019-01-26 06:09 (updated 2024-03-27 13:37)

Shotgun is a job management tool that streamlines file syncing and job submissions to HPC systems.

Important notes on HPC usage

Jobs not requiring a full node

IMPORTANT: Please read and understand the following even if it doesn’t appear relevant to your current workflow — this problem has happened more than once and burned our HPC allocation 3x — 30x faster than our actual usage.

For all queues on KyRIC, Stampede2, and the default queues on Anvil (standard), Bridges2 (RM), Expanse (compute), Perlmutter (--qos regular), and Cori (regular), you cannot use a fraction of a node. For example, even if you request 48 CPUs on Bridges2, we will be charged the full 128 CPUs because the remaining CPUs on that node cannot be used by other users. Similarly, if you request 4 node with 48 CPUs per node, we will be charged for all 128 CPUs on all 4 nodes. Therefore:

If you need to use less than a full node, use Darwin, Rockfish, or shared queues on those HPC systems that have them: Anvil (shared), Bridges2 (RM-shared), Expanse (shared), Perlmutter (--qos shared), Cori (shared). Typically, you can request up to half of all CPUs on a single node on these queues. Beyond that you will need to use the regular queues. Please double check the linked HPC documentation for details.
If you need to use more than one node, such as with a NEB job, make sure to select the number of CPUs per image such that the total CPUs required for the entire NEB job result in an integer multiple of nodes.
Check your global shotgun configuration file to ensure the default submission queue is the shared queue for each HPC that has one. This helps to avoid running single or half node jobs on compute or regular queues in case the queue (-q) tag is not specified during shotgun submit.
In general, any submitted jobs are returned with the QOS error if that system’s allocations are exhausted except on GHPCC. It’s important to notice if GHPCC resources are exhausted since the overcharge on jobs exceeding our allocation is high.

Parallelization

A node is a physical computer unit that contains a motherboard, a power supply, network cards, etc. A node may contain several sockets, which a CPU chip (like this) plugs into. A modern CPU usually contains more than one cores, sometimes referred to interchangeably as CPUs or physical CPUs when talking about parallelization. A CPU core may contain one or two hardware threads (hyper-threading), which makes N physical CPU cores appear as 2N logical cores. See below on when hyper-threading is useful and why it is often disabled in HPC centers.

When a program runs, a process is created that has its own memory space. A process may spawn several threads (different from hardware threads above) that share the same memory space. Because of memory sharing, OpenMP or other thread-based parallelization typically only works within a node. MPI is a message-passing technology to communicate between processes. MPI therefore works for processes running on the same node as well as across multiple nodes. For compute-bound programs, the total number of threads ([number of processes] × [number of threads per process]) should equal the number of physical CPU cores available to the program. A program may use both MPI and OpenMP for better parallelization, since OpenMP threads can be more memory efficient. However, it is important to note that none of these are automatic. Parallelization requires changes to the code and what part/functionality of a program has been parallelized and using what technique depends on the implementation. Make sure to refer to the software documentation and benchmark for your own systems.

Shotgun does not deal with the thread/process division. It only asks the queuing system for permission to use X number of CPU cores. If we want to use hydrid MPI/OpenMP, we can do so in the submission script, which can be something like (for GROMACS)

nthreadperrank=4  # choose according to benchmark results
nrankperjob=$((nprocperjob/nthreadperproc))  # make sure it is divisible
$MPIRUN $nrankperjob $PROG -ntomp $nthreadperrank ... (remaining arguments)

Hyper-threading

Some HPC systems (e.g., NERSC Perlmutter and Cori, and many Unity nodes) have hyper-threading enabled (i.e., a physical CPU core appears to be two logical CPU cores). Hyper-threading is useful for programs that are IO-bound. Most of our workloads are instead CPU-bound, so running two processes on the two logical CPU cores means that the two processes would take turns to run on the same underlying physical CPU. This is very inefficient. So we should always run jobs according to the number of physical CPU cores we need (i.e., if you need to use one physical CPU, request two CPUs on these systems: shotgun submit -c 2). The SLURM code snippet /home/common/sample-input/util-slurm.sh used by the submission scripts has been updated to divide the number of CPUs the scheduler allocates by the number of hyper-threads (c.f., nprocpernode=$((SLURM_CPUS_ON_NODE/hyperthreading))). If you have written your own Unity submission scripts, consider making similar changes.

Previously, the Shotgun parameter cpus_per_node was set to the number of physical CPUs for Cori, while the submission script did not account for hyper-threading. This would work fine for the regular queues on Perlmutter and Cori, but would be problematic for their shared queues.
Similarly, for Unity, we used shotgun submit --extra-options='--exclusive' to deal with hyper-threading, but this only worked for jobs that require full nodes (e.g., VASP).
The current approach works for both full and partial node situations. However, for partial nodes, it assumes that the scheduler will allocate logical CPUs that belong to the same physical CPUs first (and not one logical CPU from physical CPU 1 and and another logical CPU from physical CPU 2). So if you are not using full nodes, please observe code performance.
The file /home/common/sample-input/global-shotgun.yaml has been updated to distinguish the different types of machines Unity has. Please update your ~/.shotgun.yaml accordingly. For example, unity-40Cx2 means that the system has 40 physical CPUs and 2 hyper-threads per physical CPU (i.e., a total of 80 CPUs per node can be requested shotgun submit -c 80).

Checking Allocation Balances

Most HPCs use SLURM as the job scheduler, so commands like squeue -u $USER and scontrol show node a395 work on multiple HPCs like Unity, Expanse, Anvil, Perlmutter, etc. We can sign into ACCESS and NERSC Iris web portals to see all allocations at once. Stampede3 has an additional TACC User Portal. Finally, we can also check allocations on the HPC directly, but the method varies depending on machine.

mybalance  # Anvil
accounts  # Delta
module add sdsc; expanse-client project umn127  # Expanse
/usr/local/etc/taccinfo  # Stampede3
iris  # NERSC Perlmutter

Set up shotgun on group server

This part is done on the group server, so SSH (PuTTY for Windows) there first before proceeding.
Initialize the global settings:
```
shotgun init
```
It generates a file ~/.shotgun.yaml. You should then edit this file following the example here.
To prepare a HPC server (e.g., login.hpc.example.org) for use as a remote target in shotgun:
- Add your public key to ~/.ssh/authorized_keys on the HPC server. You will need to SSH (PuTTY for Windows) to the HPC server and edit that file. Depending on where you have done the first step in SSH-Agent, you public key is stored in ~/.ssh/id_rsa.pub either on your workstation or the group server. Also consult the instructions for copying the public key under SSH-Agent.
- If the HPC server does not allow public key authentication, you will want to keep the SSH session open as long as possible by adding a ControlPersist parameter to the corresponding Host in ~/.ssh/config:
```
Host hostA
  HostName hostA.dept.univ.edu
  User userX
  ControlPersist 168h
```
More examples are available for XSEDE, ALCF, NERSC, EMSL, MGHPCC machines.

Basic workflow

When we start a new project, create a folder on the group server, $HOME/projects/ABC, which we call local_job_root, and initialize the project settings:
```
mkdir $HOME/projects/ABC && cd $HOME/projects/ABC
shotgun new-project
```
This generates a file $HOME/projects/ABC/shotgun.yaml, which we should edit following the example here.
We may create a directory hierarchy for different molecules, at different temperatures, and multiple independent runs for each temperature; e.g., $HOME/projects/ABC/moleculeX/123K/RUN1. In each job directory, we prepare all the necessary input files and job scripts. A good practice here is to put shared files that should be the same across all job directories (e.g., common settings, forcefield parameters, job scripts) to a central location (i.e., local_asset_dir), say $HOME/projects/ABC/bin and create symbolic links in each job directory:
```
ln -sf ../../bin/sub-gromacs.script .  # Run in the job directory $HOME/projects/ABC/moleculeX/123K/RUN1
```
NOTE: these symlinks must use relative paths instead of absolute paths, for reasons that will become clear shortly.
We can now submit the job:
```
shotgun submit -c 1 -m 5200 -n 1 -t 24 -r bridges -q RM-shared -j ../../bin/sub-gromacs-bridges.script  # submitting the job
```
NOTE: the ./ part in ./sub-gromacs-bridges.script is required. See shotgun --help submit for a list of options that can be used.
IMPORTANT TO UNDERSTAND: When we do shotgun submit, all files in the current working directory ($HOME/projects/ABC/moleculeX/123K/RUN1 are rsync-ed to /pylonX/GROUP/USER/projects/ABC/moleculeX/123K/RUN1 on bridges (using the sample setting below, this is moleculeX/123K/RUN1 inside job_root you specified in $HOME/projects/ABC/shotgun.yaml). Shotgun also syncs everything in $HOME/projects/ABC/bin to /pylonX/GROUP/USER/projects/ABC/bin (i.e., local_asset_dir to asset_dir on bridges).
- It is important to understand that eventually the job script will be run on the target HPC system (e.g., bridges). Therefore, if we had created the symblinks using absolute paths:
```
ln -sf $HOME/projects/ABC/bin/sub-gromacs.script .  # Run in the job directory $HOME/projects/ABC/moleculeX/123K/RUN1
```
When files are transferred, these symlinks will become broken, because on bridges, files such as sub-gromacs.script do not exist at the location pointed to.
- Another important point is the line: BASEDIR="$HOME/tools/gromacs-2019.3/gcc-double-impi-fftw" in the job script sub-gromacs-bridges.script. This line species the location of the Gromacs executable and is outside asset_dir. The rationale is that these executables are large and likely used by many different projects, so we place them in a shared location that is not synced every time we do shotgun submit. We therefore have to rsync / scp whatever executables you need to the HPC server.
```
rsync -avzP /home/common/GROMACS/bridges/gromacs-2019.3-gcc-double-impi-fftw.tar.bz2 bridges:.
ssh bridges 'mkdir -p ~/tools/gromacs-2019.3; tar xf ~/gromacs-2019.3-gcc-double-impi-fftw.tar.bz2 -C ~/tools/gromacs-2019.3 && rm -f ~/gromacs-2019.3-gcc-double-impi-fftw.tar.bz2'
```
On the other hand, absolute paths that do not point to your own directories (e.g., BASEDIR="/home/user/tools/gromacs-2019.6/gcc-double-impi-fftw" in sub-gromacs-expanse.script or BASEDIR="/global/common/software/m3393/tools/gromacs-2019.3/intel-double-impi-mkl-knl" in sub-gromacs-bridges.script) are group accessible. We can just verify that the executable exists and, if so, we do not need to rsync anything.
We can check for the progress of the calculations using either
```
shotgun check-this -c -s  # Run in the job directory $HOME/projects/ABC/moleculeX/123K/RUN1
```
or
```
shotgun check-all -c -s  # Can be run anywhere
```
The first sub-command checks the status of all submitted jobs and can be run anywhere, and the second sub-command checks the job in the current directory (e.g., $HOME/projects/ABC/moleculeX/123K/RUN1). The flag -c cancels all jobs except the first one running (see the explanation below), and -s downloads the files.

NOTE: Sometimes ssh fails silently and you may need to run shotgun check-* a second time. If for whatever reason, shotgun check-* marks a job completed/exited but fails to sync the files back, you can force a download using
```
shotgun download -t bridges  # Run in the job directory $HOME/projects/ABC/moleculeX/123K/RUN1
```

Bundle jobs

Some HPC systems allow only jobs using full nodes (thus no single CPU jobs) or favor jobs requesting a large number of nodes. For these HPC systems, we can bundle separate calculations into a meta job. Remember that we submit machine-specific scripts such as sub-gromacs-nersc.script to the corresponding HPC system to set up OS environment (e.g., loading modules, figuring out available compute nodes), but the script will pass the actual job execution to sub-gromacs.script that is machine agnostic. To bundle jobs, we replace sub-gromacs-nersc.script with a script we will call qbatch-gromacs-nersc.script that similarly sets up the environment but calls sub-gromacs.script from the directories of all jobs that we want to include in a single job submission. Using the example above, the steps for bundling jobs are outlined below.

Generate a text file listing the individual job directories: $HOME/projects/ABC/moleculeX/123K/RUN1, $HOME/projects/ABC/moleculeX/123K/RUN2, $HOME/projects/ABC/moleculeX/133K/RUN1, etc. You can generate this list by
```
cd $HOME/projects/ABC
ls -d -l */*K/RUN* | gawk '{print $NF}' > jobs.txt
```
or if the job directories are not structured systematically:
```
cd $HOME/projects/ABC
find . -iname 'sub-gromacs.script' -execdir pwd \; > jobs.txt
```
Prepare a template submission script using the corresponding sub-gromacs-ABC.script as a base line. You can compare sub-gromacs-nersc.script and qbatch-gromacs-nersc.script in /home/common/sample-input/gromacs/bin to get an idea of what changes need to be made. For this sample file, all jobs are run simultaneously, and source sub-gromacs.script in sub-gromacs-nersc.script is replace by the code below.
```
source util.sh
export nprocperjob=2  # !!!change this according to how many CPUs you want to use per calculation!!!
((njobspernode=nprocpernode/nprocperjob))
nprocpernode=$nprocperjob
export MPIRUN PROG HOSTS nprocpernode

inode=-1
ijob=0

#>>>BEGIN<<<
cd "$ROOT/$job_dir"
[ $((ijob++ % njobspernode)) -eq 0 ] && ((inode++))
HOSTS=${NODE[inode]} MPIRUN="srun -w ${NODE[inode]} -c 1 --cpu_bind=cores -n" ./sub-gromacs.script 2>stderr 1>stdout &

#>>>END<<<

wait
```
Shotgun will create the section marked by #>>>BEGIN<<< and #>>>END<<< for each job directory and replace $job_dir accordingly. It is up to you to make sure that the template submission script does what you want and is compatible with sub-gromacs.script. For example, if you want to run jobs one by one, starting a new one only after the previous job finishes, then you would use the following instead.
```
source util.sh
export MPIRUN PROG HOSTS nprocpernode

#>>>BEGIN<<<
cd "$ROOT/$job_dir"
# The command below should be blocking; i.e., not having ampersand at the end
./sub-gromacs.script 2>stderr 1>stdout

#>>>END<<<
```
Create an actual submission script from the template and the job listing. We also need to create symbolic links to the util-slurm.sh files since they are expected to exist in the same directory as the submission script.
```
cd $HOME/projects/ABC
shotgun bundle -j ../bin/qbatch-gromacs-nersc.script -d jobs.txt -o qbatch-nersc.sh
ln -sf ../bin/util-* .
```
Double check qbatch-nersc.sh before submitting the job. Say if you have a total of 48 jobs with each using 2 CPUs, since compute nodes on Bridges have 32 CPUs per node, you would then request 48 x 2 / 32 = 3 full nodes.
```
cd $HOME/projects/ABC
shotgun submit -c 32 -m 5200 -n 3 -t 24 -r cori -q regular -j ./qbatch-nersc.script
```

Cycle through a large list of jobs

If we need to bundle jobs but there are a large number of them and each job is expected to take different amounts of time, then the technique described in Bundle jobs will be wasteful as the CPUs that complete sooner will sit idle for the longest job to finish. Instead, we can write a script to continuously fetch a job from a list, run it, and once completed, fetch another job, until the list is exhausted. This strategy is most useful when individual jobs do not take too long and we can cycle through a few of them in one submission. This way we request a small number of nodes and should be able to keep them occupied (almost) at all times.

Similar to Bundle jobs, generate a text file in a subfolder called jobs/run1.txt, listing the individual job directories: $HOME/projects/ABC/moleculeX/123K/RUN1, $HOME/projects/ABC/moleculeX/123K/RUN2, $HOME/projects/ABC/moleculeX/133K/RUN1, etc. You can generate this list by
```
cd $HOME/projects/ABC
mkdir -p jobs
ls -d -l */*K/RUN* | gawk '{print $NF}' > jobs/run1.txt
```
or if the job directories are not structured systematically:
```
cd $HOME/projects/ABC
mkdir -p jobs
# Use a file to identify the job directory for jobs to be run
find . -iname 'gromacs.tpr' -execdir pwd \; > jobs/run1.txt
```
Prepare the cycle submission script. An example can be found here: /home/common/sample-input/topmon/bin/qcycle-topmon-unity.script. This script will fetch jobs from jobs/run1.txt, jobs/run2.txt, etc., call qcycle-topmon.script to perform the actual calculation, and record completed jobs in logs/JOBNAME.done. Running jobs will have a file running.lock in its directory to signal other idle CPUs not to launch this job.
Say you have a total of 400 jobs, each using 1 CPUs, and if you want to target two unity-40C nodes, you would request two nodes with the qcycle-topmon-unity.script script:
```
cd $HOME/projects/ABC
shotgun submit -n 2 -t 24 -r unity-40C -j ./qcycle-topmon-unity.script
```
Since each node requested will have 40 CPUs, 80 jobs will run at the same time and on average each CPU will run 5 jobs before all jobs finish.

Summary of available commands

shotgun submit -c 8 -m 5200 -n 1 -t 24 -r bridges -q RM-shared -j ../../bin/sub-gromacs-bridges.script
shotgun check-all [-c] [-s] # checks the status of all submitted jobs; can be run anywhere
shotgun check-this [-c] [-s] # checks the job in the current directory. -c cancels all jobs except the first one running, -s downloads the files
shotgun download -t bridges  # just download files
shotgun cancel-job -t bridges
shotgun hold -t bridges
shotgun release -t bridges

*/30 * * * * bash -l -c 'shotgun check-all --sync --cancel 2>> ~/shotgun.err >> ~/shotgun.out'

Shotgun mode

If we really want our jobs to start as soon as possible but all the HPC machines have full queues, we can submit a single job to multiple HPC machines (e.g., Bridges2, Unity, Expanse), then check-all and check-this as frequently as reasonable and possible with the -c flag. This flag will keeping only the job on the machine that starts running the earliest and cancel jobs submitted to all other machines.

Shotgun configuration files

~/.shotgun.yaml is the global configuration file, while /path/to/projects/ABC/shotgun.yaml is the per-project configuration and will override what’s in ~/.shotgun.yaml if they both contain the same setting. Number of nodes, memory, etc. that you specify on the command line have the highest priority, and will override both configuration files.

Sample global configuration

Location: /home/common/sample-input/global-shotgun.yaml
The entry cori-knl below has a key host. This can be a full hostname (e.g., cori.nersc.gov) or a Host defined in ~/.ssh/config, which is the case here. All the other remote targets do not have an explicit host key, in which case it defaults to the name of the remote target; i.e., host for the ghpcc06 entry defaults to ghpcc06. Therefore make sure to set up ~/.ssh/config accordingly so that ssh ghpcc06 etc. can run correctly on the terminal.
Edit the default queue for each host to be the “shared” queue. This way, jobs will not run on compute nodes if the queue tag is not specified during job submission. This helps avoid submitting small jobs (requiring less than one full node) to queues that will charge for a full node regardless of the CPUs requested. Refer to the section above on Important Notes on HPC Usage for more details on node charging.

Sample project configuration

Location: /home/common/sample-input/project-shotgun.yaml
Certain files such as wavefunctions generated by the calculations are very large and often not needed. You can exclude these files using download_excludes. Wildcards are allowed.
upload_excludes excludes files for uploads. You should always keep job.yaml in this list.
local_asset_dir contains all the common scripts for the project and is synced to asset_dir on the remote target. When local_asset_dir is absent, the asset syncing is skipped.
job_root is the project root directory on the HPC machine that should map to the project root directory on the group server that contains the shotgun.yaml file. You should consult the User Guides for the target HPC machine (look for headings such as Storage, File Systems, File Spaces) and select a fast filesystem. As an example, for Bridges2, $PROJECT is the file system we should use. You can find out what the environment variable $PROJECT points to by entering echo $PROJECT at the terminal after SSH-ing to Bridges2.
In this example, the three NERSC machines (cori, cori-knl, and edison) share the same file system, but you will typically want to use different job_root directories so that their output do not override each other. This is achieved simply by adding a suffix to the directory name.

How to troubleshoot failed jobs

First, look at the job stdout/stderr, as well as the screen output redirected in submission scripts (e.g., sub-vasp.script redirects to vaspout/vasperr) etc. to figure out whether:

The jobs just exited without going to a compute node, or
They started on a compute node but the script decided to quit (e.g., if you have a DONE file, the sub-vasp.script will not rerun the calculation), or
VASP didn’t run or error exited.

For some of these problems, it may be helpful to SSH to the target HPC machine directly and examine the job status with

squeue -u $USER
scontrol show job JOB_ID

If the job appears to have started but the output files do not reveal enough clues. Continue troubleshooting with an interactive job:

SSH to the HPC machine and cd to the job directory. You can find out where the remote job directory is from the output of shotgun submit:
```
[bridges2] Syncing (LOCAL) /home/USER/projects/ABC/... -> (REMOTE) /ocean/projects/GROUP/USER/projects/ABC/...
```
First check if there are any extra files that are not downloaded back;
Request an interactive job, something like: srun --nodes=1 --ntasks-per-node=128 --cpus-per-task=1 --time=1:00:00 --partition compute --account umn127 --pty bash. You can figure out the correct command to use by taking the last line of output from shotgun submit, for example
```
[bridges2] Submitting: sbatch --nodes=1 --ntasks-per-node=128 --cpus-per-task=1 --time=1:00:00 --partition compute --account umn127 ./sub-vasp-bridges2.script
```
Change sbatch to srun and ./sub-vasp-bridges2.script to --pty bash at the end;
Copy/paste to run the lines module add ..., export ..., MPIRUN=..., PROG=... from sub-vasp-bridges2.script to set up environment in this interactive session;
Finally, call vasp directly:
```
$MPIRUN $((SLURM_NTASKS_PER_NODE*SLURM_CPUS_PER_TASK)) "${PROG}"
```
You can figure out the command to use by adapting a line you can find in sub-vasp.script:
```
/usr/bin/time -ao $FILELOG $MPIRUN $nprocperjob "${PROG}${gamma}" 1>vaspout 2> vasperr
```
This line uses a few additional variables the submission scripts sets up and we have simplified it in the example code above.

Notes on Unity usage

Unity has a heterogeneous set of nodes. We can log into Unity and check the job status.

ssh unity
sinfo

For example, the screenshot shows nodes 16 — 20 to be available. Consulting Unity node list, we find that these nodes have 2 of Intel Xeon Gold 6148 CPUs, for a total of 2 x 20 = 40 physical CPUs. We can then target these nodes by submitting to the unity-40C queue.

A common error when submitting jobs is invalid node specification. On Unity, the node specs/constraints are changed often and without notice, so we can just check the current specs when this error occurs. The global ~/.shotgun file is commented with node identifiers, so we can easily use these names check the specs on Unity and match the Active Features with the constraints listed in ~/.shotgun.

#on Unity
scontrol show node ceewater-cpu001 | grep ActiveFeatures | cut -d '=' -f 2
#on group server where $specs is above output
sed -i "/unity-preempt-24C/{n; n; n; n; n; n; /constraint/:.*/: $specs/;}" ~/.shotgun.yaml

Unity is unique in that job submission limits are shared across the group instead of per user (max CPU limit = 1000 for the whole group). To get around this, we can submit jobs to the preempt queues which don’t factor into this CPU limit. We can also check the group’s CPU usage in real-time using unity-slurm-account-usage.