Compilation & Tuning

(updated )

Comments

HPC machine microarchitecture

  • ACCESS NCSA
    • Delta (AVX2): 2.45GHz 128 core, 2GB/core: 2 x AMD EPYC 7763 “Milan” (Zen 3, 32MB L3), 256GB
    • Delta-GPU (AVX2; cc86): 4 x NVIDIA A40 (48Gb), 2.45GHz 64 core, 4GB/core: AMD EPYC 7763 “Milan”, 256GB
    • Delta-GPU (AVX2; cc80): 4 x NVIDIA A100 (40Gb), 2.45GHz 64 core, 4GB/core: AMD EPYC 7763 “Milan”, 256GB
    • Delta-GPU (AVX2; cc80): 8 x NVIDIA A100 (40Gb), 2.45GHz 128 core, 16GB/core: 2 x AMD EPYC 7763 “Milan”, 2048GB
    • Delta-GPU (AVX2; CDNA): 8 x AMD MI100 (32Gb), 2.45GHz 128 core, 16GB/core: 2 x AMD EPYC 7763 “Milan”, 2048GB
  • ACCESS JHU
    • Rockfish (AVX512): 3.0GHz 48 core, 4GB/core: 2 x Intel Xeon 6248R (Cascade Lake), 192GB DDR4
    • Rockfish-GPU (AVX512): 4 x NVIDIA A110 (40Gb), 3.0GHz 48 core, 4GB/core: 2 x Intel Xeon 6248R (Cascade Lake), 192GB DDR4
  • ACCESS Purdue
    • Anvil (AVX2): 2.45-3.5GHz 128 core, 2GB/core: 2 x AMD EPYC 7763 (Zen 3), 256GB DDR4, 56 Gb/s Interconnect
    • Anvil-GPU (AVX2; cc80): 4 x NVIDIA A100, 2.45-3.5GHz 128 core, 4GB/core: 2 x AMD EPYC 7763 (Zen 3), 512GB DDR4, 56 Gb/s Interconnect
  • ACCESS PSC
    • Bridges2 (AVX2): 2.25-3.40GHz 128 core, 2GB/core: 2 x AMD EPYC 7742 (Zen 2, 256MB L3), 256GB DDR4, Mellanox ConnectX-6-HDR Infiniband 200Gb/s Adapter
    • Bridges2-GPU (AVX512; cc70): 8 x NVIDIA Tesla V100 32GB SXM2, 2.5-3.9GHz 40 core, 12.8GB/core: 2 x Intel Xeon Gold 6248 (Cascade Lake), 512GB DDR4, Mellanox ConnectX-6-HDR Infiniband 200Gb/s Adapter
    • Bridges2-AI-1 (AVX512; cc70): 8 x NVIDIA V100 16GB, 2.4-3.7GHz 40 core, 4.8GB/core: 2 x Intel Xeon Gold 6148 (Skylake), 192GB DDR4, Mellanox ConnectX-6-HDR Infiniband 200Gb/s Adapter
    • Bridges2-AI-2 (AVX512; cc70): 16 x NVIDIA Volta V100 32GB, 2.7-3.7GHz 48 core, 32GB/core: 2 x Intel Xeon Platinum 8168 (Skylake), 1.5TB DDR4, Mellanox ConnectX-6-HDR Infiniband 200Gb/s Adapter
    • Bridges (AVX2): 2.3/3.3GHz 28 core, 4.57GB/core: 2 x Intel Xeon E5-2695v3 (Haswell, 35MB LLC), 128GB DDR4
  • ACCESS SDSC
    • Expanse (AVX2): 2.25-3.40GHz 128 core, 2GB/core: 2 x AMD EPYC 7742 (Zen 2, 256MB L3), 256GB DDR4, 56 Gb/s Interconnect
    • Expanse-GPU (AVX512; cc70): 4 x NVIDIA V100 SMX2, 2.5-3.9GHz 40 core, 9.6GB/core: 2 x Intel Xeon Gold 6248 (Cascade Lake), 384GB DDR4
    • Comet (AVX2): 2.5GHz 24 core, 5.33GB/core: 2 x Intel Xeon E5-2680v3 (Haswell), 128GB DDR4, 56Gbps FDR InfiniBand with rack-level full bisection bandwidth and 4:1 oversubscription cross-rack bandwidth
  • ACCESS UDel
    • Darwin (AVX2): 2.5-3.35GHz 64 core, 8GB/core: 2 x AMD EPYC 7502 (Zen 2, 128MB L3), 512GB, Mellanox HDR100
    • Darwin-GPU: NVIDIA T4 (cc75) / NVIDIA V100 (cc70) / AMD MI50
  • ACCESS UTexas
    • Stampede3
      • SPR (AVX512): 1.90-3.50GHz 112 core 1.14GB/core: 2 x Intel Xeon CPU MAX 9480 (“Sapphire Rapids HBM, 48KB L1 + 1MB L2 + 112.5MB L3 per socket), 128 GB HBM 2e, 100Gb/sec Intel Omni-Path (OPA) network (interconnected with PVC nodes)
      • PVC (AVX512; Xe-HPC): CPU: 2.10-3.70GHz 112 core 4.57GB/core: 2 x Xeon CPU MAX 8480 (Sapphire Rapids, 48KB L1 + 1MB L2 + 112.5MB L3 per socket), 512 HB DDR5. GPU: 900-1600 MHz 512 core 1GB/core: 4 x Intel Data Center GPU Max 1550s, 100Gb/sec Intel Omni-Path (OPA) network (interconnected with SPR nodes)
      • ICX (AVX512): 2.3GHz 80 core 3.2GB/core: 2 x Intel Xeon Platinum 8380 (IceLake, 48KB L1 + 1.25MB L2 + 60MB L3 per socket), 256GB DDR4 3.2GHz, 100Gb/sec Intel Omni-Path (OPA) network
      • SKX (AVX512): 2.1GHz 48 core 4GB/core: 2 x Intel Xeon Platinum 8160 (Skylake, 32KB L1 + 1MB L2 + 33MB L3 per socket), 192GB DDR4 2.67GHz, 100Gb/sec Intel Omni-Path (OPA) network
    • Stampede2 (multi-architecture: -xCOMMON-AVX512 -axCORE-AVX512,MIC-AVX512)
      • KNL (MIC-AVX512): 1.4GHz 68 core 1.4GB/core: Intel Xeon Phi 7250 (Knights Landing, 32KB L1 + 1MB L2 per two core + 16GB MCDRAM), 96GB DDR4 2400MHz
  • NERSC
    • Perlmutter (AVX2): 2.45-3.5GHz 128 core, 4GB/core: 2 x AMD EPYC 7763 (Zen 3), 512GB DDR4, Slingshot 11 Interconnect
    • Perlmutter-GPU (AVX2; cc80): 4 x NVIDIA A100, 2.45-3.5GHz 64 core, 4GB/core: AMD EPYC 7763 (Zen 3), 256GB DDR4, Slingshot 11 Interconnect
    • Cori KNL (MIC-AVX512): 2.3GHz 32 core 4GB/core: 2 x Intel Xeon E5-2698v3 (Haswell, 32KB L1 + 256 KB L2 + 40MB L3 per socket), 128 GB DDR4 2133 MHz, Cray Aries with Dragonfly topology
    • Cori (AVX2): 1.4GHz 68 core 1.4GB/core: Intel Xeon Phi 7250 (Knights Landing, 32KB L1 + 1MB L2 per two core + 16GB MCDRAM), 96 GB DDR4 2400 MHz
    • Edison (AVX): 2.4GHz 24 core, 2.67GB/core: 2 x Intel E5-2695v2 (Ivy Bridge), 64GB DDR3 1866MHz, Cray Aries dragonfly topology
  • ANL
    • Mira: IBM Blue Gene/Q
  • UMass
    • Unity (multi-architecture: -xCORE-AVX2 -axCORE-AVX512)
      • Intel Skylake
      • AMD Zen 2
      • GPU: GTX Titan X (cc52), GTX 1080ti (cc61), RTX 2080 (cc75), RTX 2080ti (cc75), Quadro RTX 8000 (cc75), Tesla M40 (cc52), Tesla V100 (cc70), Tesla A100 (cc80)
    • GHPCC
      • Intel Skylake -> AVX512
      • Haswell -> use Mesabi builds
      • Sandy Bridge -> Itasca_sb builds
      • AMD Piledriver, Bulldozer -> Itasca Nehalem builds
  • Others
    • Enterprise (AVX2): 2.2GHz 24 core, 2.67GB/core: 2 x Intel Xeon E5-2650v4 (Broadwell), 64GB DDR4
    • Mesabi (AVX2): 2.5GHz 24 core, 2.67GB/core: 2 x Intel Xeon E5-2680v3 (Haswell), 64GB DDR4
    • Metropolis (AVX)
      • 2.4GHz 8 core, 2GB/core: 2 x Intel Xeon E5-2609 (Sandy Bridge), 16GB
      • 2.5GHz 8 core, 2GB/core: 2 x Intel Xeon E5-2609v2 (Ivy Bridge), 16GB
      • 2.1GHz 8 core, 2GB/core: 2 x Intel Xeon E5-2620v2 (Ivy Bridge), 16GB
      • 2.1GHz 16 core: 2 x Intel Xeon E5-2620v4 (Broadwell)
    • Itasca
      • _sb suffix (AVX): Intel Sandy Bridge
      • Regular (SSE4.1): Intel Nehalem
    • 4170HE: AMD K10

Compiler flags

  • CP2K, 7.1 H20-256

AOCC+AOCL: -O3 -march=znver3 -ffast-math -fopenmp -ffree-form -Hx,47,0x10000008 -Mbackslash

  • VASP, 6.1.2 GaAsBi-64 (VASP guide refers to -02 and not -O3 as default optimisation level)

Intel: -O2 -assume byterecl -w -march=core-avx2 -fma -mavx2 -mfma -free -names lowercase -mkl=sequential -qopenmp -lmkl_blacs_openmpi_lp64 -lmkl_scalapack_lp64

  • Quantum Espresso, 6.5 AUSURF112

GCC+AOCL: -O3 -march=znver2 -ffast-math -fopenmp

  • NAMD, 2.14

AOCC+AOCL: -optimize -production -DCMK_OPTIMIZE -O3 -Ofast -g -m64 -march=znver3 -fPIC -ffp-contract=fast -fopenmp -Wno-deprecated-register -Wno-non-c-typedef-for-linkage -mllvm -inline-threshold=3000 -finline-hint-functions -mllvm -inlinehint-threshold=10000 -finline-functions

  • WRF, 3.9.1.1, CONUS 2.5KM

AOCC: -march=znver3 -ffp-contract=fast -mavx2 -mfma -mprefer-vector-width=256 -Ofast -ffast-math -fopenmp -m64 -w -finline-aggressive -fPIC

  • OpenFOAM, 2012, Motorbike 42m cells

GCC: -O3 -march=znver2 -mfma -ffp-contract=fast -funroll-loops -fno-signed-zeros -fno-trapping-math -fno-math-errno -ffinite-math-only -fno-rounding-math -fno-signaling-nans -fcx-limited-range -fexcess-precision=fast

Generate auto-dispatch code paths with Intel compilers

Some HPC systems (e.g., NERSC Cori, XSEDE Stampede2) have different kinds of compute nodes with different instruction sets: Broadwell (-xCORE-AVX2), Skylake/Icelake (-xCORE-AVX512, and KNL (-xMIC-AVX512).

A simpler approach can be used when building with the Intel compiler: use CPU auto dispatch to build a multi-architecture (“fat”) binary that contains alternate code paths with optimized vector code for each type of compute nodes. To produce a multi-architecture binary for Haswell, Skylake/Icelake, and KNL nodes, build with the following options -xCORE-AVX2 -axCORE-AVX512,MIC-AVX512.

These particular choices allow you to build on any node and use CPU dispatch to produce a multi-architecture binary. We recommend that you specify these flags in both the compile and link steps. Specify an optimization level (e.g. -O3) along with any other appropriate compiler switches: icc -xCORE-AVX2 -axCORE-AVX512,MIC-AVX512 -O3 mycode.c -o myexe.

The -x option is the target base architecture (instruction set). The base instruction set must run on all targeted processors. Here we specify CORE-AVX2, which is native for older Broadwell processors and supported on all KNL and Skylake/Icelake nodes. This option allows configure scripts and similar build systems to run test executables on any login or compute node. The -ax option is a comma-separated list of alternate instruction sets: CORE-AVX512 for Skylake and Icelake, and MIC-AVX512 for KNL.

If compatibility with Broadwell nodes is not needed, more aggressive base option is -xCOMMON-AVX512, which is a subset of AVX512 that runs on all KNL, Skylate, and Icelake nodes.

On heterogeneous HPC clusters, it’s best to avoid building with -xHost (a flag that means “optimize for the architecture on which I’m compiling now”). Using -xHost on a Skylake node, for example, will result in a binary that won’t run on KNL.

Don’t skip the -x flag in a multi-architecture build: the default is the very old SSE2 (Pentium 4) instruction set. Don’t create a multi-architecture build with a base option of either -xMIC-AVX512 (native on KNL) or -xCORE-AVX512 (native on Skylake/Icelake); there are no meaningful, compatible alternate (-ax) instruction sets:

$ icc -xCORE-AVX512 -axMIC-AVX512 -O3 mycode.c -o myexe  # WRONG! Base incompatible with alternate

Simplicity is a major advantage of this multi-architecture approach: it allows you to build and run on any nodes, and performance is generally comparable to single-architecture builds. There are some trade-offs to consider, however. This approach will take a little longer to compile than single-architecture builds, and will produce a larger binary. In some cases, you might also pay a small performance penalty over single-architecture approaches. For more information see the Intel documentation.

Auto-dispatch for both Intel and AMD CPUs

For HPC systems (e.g., Unity) with both AMD and Intel compute nodes, we may build auto-dispatch executables with -march=core-avx2 -mtune=core-avx2 -axCORE-AVX512. See the notes below for more details.

Intel compilers for AMD

The Intel compiler is developed and targeted for the Intel hardware and hence it has some minor issues when using it with AMD hardware. Execution of code compiled with the Intel compiler might lead to crash when executed on the AMD EPYC CPUs. The recommended flag to use the AVX2 instructions supported by the AMD EPYC CPUs is -march=core-avx2 -mtune=core-avx2.

In some cases, when compiling with the -xavx, -xavx2 or -xcore-avx2 flags your application will abort with a message that the processor is not compatible:

Please verify that both the operating system and the proce_ssor support Intel(R) X87, CMOV, MMX, FXSAVE, SSE, SSE2, SSE3, SSSE3, SSE4_1, SSE4_2, MOVBE, POPCNT, AVX, F16C, FMA, BMI, LZCNT and AVX2 instructions.

To solve this problem, avoid using the above-mentioned flags for the compilation of the main() function. Avoiding the use of Interprocedural Optimization (remove the -ipo flag) may also be tried.

Building notes

NOTE: These are abbreviated notes that have worked in the past. Always read the current documentation, and only use these notes as hints and examples for putting together a working build.

LAMMPS

Official Documentation

CMake procedure

module add intel/2015/update3; module add impi/5.0.2.044/intel
mkdir ~/tools/lammps-git/build
cd ~/tools/lammps-git/build
cmake -DCMAKE_CXX_COMPILER=mpiicpc -DCMAKE_C_COMPILER=mpiicc -DCMAKE_Fortran_COMPILER=mpiifort -DCMAKE_TUNE_FLAGS='-O3 -xHost -ipo -fp-model fast=2 -no-prec-div -qoverride-limits -diag-disable=10441 -diag-disable=2196' -DBUILD_MPI=yes -DBUILD_OMP=yes -DFFT=MKL -DLAMMPS_MEMALIGN=64 -DBUILD_SHARED_LIBS=on -DLAMMPS_EXCEPTIONS=yes -DPKG_COLVARS=on -DPKG_INTEL=on -DPKG_LEPTON=on -DPKG_KOKKOS=on -DPKG_KSPACE=yes -DPKG_MOLECULE=yes -DPKG_EXTRA-MOLECULE=on -DPKG_OPENMP=on -DPKG_OPT=on -DPKG_PYTHON=on -DPKG_REAXFF=on -DPKG_REPLICA=on -DPKG_RIGID=on -DCMAKE_INSTALL_PREFIX=$HOME/tools/lammps/intel-impi-mkl ../cmake
make -j 2

Example cmake line for GCC+

cmake -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpif90 -DCMAKE_TUNE_FLAGS='-O3 -mtune=native -march=native -ffast-math -fassociative-math -freciprocal-math -fno-signed-zeros -fno-trapping-math -funroll-loops' -DBUILD_MPI=yes -DBUILD_OMP=yes -DFFT=FFTW3 -DLAMMPS_MEMALIGN=64 -DBUILD_SHARED_LIBS=on -DLAMMPS_EXCEPTIONS=yes -DPKG_COLVARS=on -DPKG_INTEL=on -DPKG_LEPTON=on -DPKG_KOKKOS=on -DPKG_KSPACE=yes -DPKG_MOLECULE=yes -DPKG_EXTRA-MOLECULE=on -DPKG_OPENMP=on -DPKG_OPT=on -DPKG_PYTHON=on -DPKG_REAXFF=on -DPKG_REPLICA=on -DPKG_RIGID=on -DCMAKE_INSTALL_PREFIX=$HOME/tools/lammps/gcc-mpich2-fftw3 ../cmake

Reax (old Make system)

cd ~/tools/lammps-git/lib/reax; make -f Makefile.ifort # build Fortran reax library
cd ../../src
# Modify CCFLAGS, LINKFLAGS, FFT_INC, and FFT_LIB in MAKE/Makefile.ompi_icc to add optimizations and point to MKL libraries
# To use FFTW3: set FFT_INC = -DFFT_FFTW3, FFT_LIB = -lfftw3, and do module load fftw/3.3-ompi-double
make yes-opt yes-reax yes-user-intel yes-user-omp yes-user-reaxc
make yes-kspace yes-manybody yes-molecule yes-opt yes-reax yes-replica yes-user-intel yes-user-omp yes-user-reaxc
make ompi_icc -j 16

AOCC+AOCL 29-OCT-20 lj

-g -O3 -march=znver2 -ffast-math -ffp-contract=fast -DNDEBG -pthread -fopenmp -std=c++11

GROMACS

Official documentation: Do not rely solely on the “Quick and dirty installation”. You most likely need to read through the entire Prerequisites section to build an executable with decent performance.

Example procedure

  • gcc-mvapich-fftw
mkdir ~/tools/gromacs-5.0.4/build
cd ~/tools/gromacs-5.0.4/build
module add gcc mvapich
cmake -DCMAKE_C_COMPILER=mpigcc -DCMAKE_CXX_COMPILER=mpigxx -DGMX_DOUBLE=ON -DGMX_MPI=ON -DGMX_FFT_LIBRARY=fftw3 -DGMX_BUILD_OWN_FFTW=ON -DCMAKE_INSTALL_PREFIX=$HOME/tools/gromacs-2016.1/gcc-double-mpi ..
make -j 16 install
  • intel-impi-mkl
module add intel/2013/update2 impi/4.1.1.036/intel mkl/11.1.1.106
cmake -DCMAKE_C_COMPILER=mpiicc -DCMAKE_CXX_COMPILER=mpiicpc -DGMX_DOUBLE=ON -DGMX_MPI=ON -DGMX_FFT_LIBRARY=mkl -DCMAKE_INSTALL_PREFIX=$HOME/tools/gromacs-5.0.4/build ..
  • intel-ompi-fftw
module add intel/2013/update2 ompi/1.6.3/intel-2013-update2
cmake -DCMAKE_C_COMPILER=mpiicc -DCMAKE_CXX_COMPILER=mpiicpc -DGMX_DOUBLE=ON -DGMX_MPI=ON -DGMX_FFT_LIBRARY=fftw3 -DGMX_BUILD_OWN_FFTW=ON -DCMAKE_INSTALL_PREFIX=$HOME/tools/gromacs-5.0.4/build ..

gcc-ompi-fftw-cuda (single-precision only for GPU)

module add gcc/4.8.1 ompi/1.7.2/gnu-4.8.1 cuda/6.0
cmake -DCMAKE_C_COMPILER=mpigcc -DCMAKE_CXX_COMPILER=mpigxx -DGMX_MPI=ON -DGMX_GPU=on -DGMX_FFT_LIBRARY=fftw3 -DGMX_BUILD_OWN_FFTW=ON -DCMAKE_INSTALL_PREFIX=$HOME/tools/gromacs-5.0.4/build ..
make -j 16 install

Cray system with Intel KNL MIC (static build)

mkdir ~/tools/gromacs-2019.3/build
cd ~/tools/gromacs-2019.3/build
cmake -DCMAKE_C_COMPILER=cc -DCMAKE_CXX_COMPILER=CC -DGMX_DOUBLE=ON -DGMX_MPI=ON -DGMX_FFT_LIBRARY=mkl -DGMX_BUILD_SHARED_EXE=off -DCMAKE_INSTALL_PREFIX=$HOME/tools/gromacs-2019.3/intel-double-impi-mkl ..
make -j 16 install
rm -r *
cmake -DCMAKE_C_COMPILER=cc -DCMAKE_CXX_COMPILER=CC -DGMX_DOUBLE=ON -DGMX_MPI=ON -DGMX_FFT_LIBRARY=mkl -DGMX_BUILD_SHARED_EXE=off -DGMX_SIMD=AVX_512_KNL -DCMAKE_INSTALL_PREFIX=$HOME/tools/gromacs-2019.3/intel-double-impi-mkl-knl ..
module swap craype-haswell craype-mic-knl
make -j 16 install

Specifying prebuilt FFTW

module load fftw/3.3-ompi-double
CC=mpicc CXX=mpiCC FFTW3_ROOT_DIR=/soft/fftw/intel-ompi/3.3-double cmake -DGMX_DOUBLE=ON -DGMX_MPI=ON -DGMX_PREFER_STATIC_LIBS=ON -DCMAKE_INSTALL_PREFIX=$HOME/tools/gromacs-4.5.5/build ..

Specifying MKL libraries (typically not needed)

export MKL=/soft/intel/mkl/10.2.7.041
CC=mpicc CXX=mpiCC cmake -DGMX_DOUBLE=ON -DGMX_MPI=ON -DGMX_THREADS=OFF -DGMX_PREFER_STATIC_LIBS=ON -DCMAKE_INSTALL_PREFIX=$HOME/tools/gromacs-4.5.5/build -DGMX_FFT_LIBRARY=mkl -DMKL_INCLUDE_DIR=${MKL}/include "-DMKL_LIBRARIES=${MKL}/lib/em64t/libmkl_intel_lp64.so;${MKL}/lib/em64t/libmkl_core.so;${MKL}/lib/em64t/libmkl_sequential.so" ..
make install -j 8

AOCC+AOCL 2020.4 water_1536

-DGMX_DOUBLE=0 -O3 -Ofast -g -m64 -march=znver3 -fPIC -ffast-math -ffp-contract=fast -Wno-deprecated-copy -Wno-error=deprecated-copy --gcc-toolchain=/usr -O3 -DNDEBUG -mavx2 -mfma -funroll-all-loops -pthread -fopenmp=libomp

Comments

Comments powered by Disqus