Example Installations on HPC Systems

Example Installations on HPC Systems#

This page provides details of our experience compiling AxiSEM3D on a number of high performance computing facilities. This is often an interesting experience and, because the systems and AxiSEM3D are both evolving there is no guarantee that the solution we found still works now. Nevertheless, the information below may be of some help to others.

Archer 2#

We can mostly compile AxiSEM3D using libraries provided on the system. However, the latest boost is (one minor version) too old so that needs to be downloaded. There is also an interesting issue with the naming scheme of metis, which is avoided by making a local soft link to the library with a more common name (otherwise CMake complains). Finally, it is important to use the gcc compiler as the Cray C++ compiler throws an ICE which we have not investigated.

These commands worked on 8-4-2022:

#!/bin/bash

PRFX=/home/n03/n03/andreww/AxiSEM-3D

cd ${PRFX}

module load PrgEnv-gnu
module load cray-fftw/3.3.8.9
module load metis/5.1.0
module load cray-hdf5/1.12.0.3
module load cray-netcdf/4.7.4.3
module load cmake/3.21.3

mkdir -p dependencies
cd dependencies

[ ! -d ./boost_1_78_0 ] && wget -c https://boostorg.jfrog.io/artifactory/main/release/1.78.0/source/boost_1_78_0.tar.bz2 -O - | tar -jx -C ./

mkdir -p ./metis/lib
ln -s /work/y07/shared/libs/core/metis/5.1.0/GNU/9.3/bin ./metis/bin
ln -s /work/y07/shared/libs/core/metis/5.1.0/GNU/9.3/include ./metis/include
ln -s /work/y07/shared/libs/core/metis/5.1.0/GNU/9.3/lib/libmetis_gnu.a ./metis/lib/libmetis.a

cd ${PRFX}

git clone https://github.com/kuangdai/AxiSEM-3D.git AxiSEM3D
git -C AxiSEM3D pull

mkdir -p build
cd build

export CRAYPE_LINK_TYPE=dynamic

rm -rf ./*

cmake -Dcxx=CC \
  -Dflags="-std=c++17 -O3 -DNDEBUG -fPIC" \
  -Deigen=/work/y07/shared/libs/core/eigen/3.4.0/include \
  -Dboost=${PRFX}/dependencies/boost_1_78_0 \
  -Dfftw=/opt/cray/pe/fftw/3.3.8.9/x86_rome \
  -Dmetis=${PRFX}/dependencies/metis \
  -Dnetcdf=/opt/cray/pe/netcdf/4.7.4.3/GNU/9.1 \
  -Dhdf5=/opt/cray/pe/hdf5/1.12.0.3/GNU/9.1 \
  -Dpar_netcdf=false \
  -Dnpol=4 \
  -Ddouble=true \
  ../AxiSEM3D/SOLVER/

make -j

ARC (University of Oxford)#

The ‘trick’ here is to mix up-to-date header only C++ libraries and link to compiled libraries from the system (using the module system and letting CMake find the link paths without our ‘help’). The default compiler is g++ (this is what mpicxx points to) and that seems to be the sensible choice. Anyway, as of 9/4/2022 this works:

#!/bin/bash

PRFX=/home/eart0526/code/AxiSEM3D_april

cd ${PRFX}

module load netCDF
module load CMake

# mpicxx points to g++, which should be okay I think

mkdir -p dependencies
cd dependencies

[ ! -d ./eigen-3.4.0 ] && \
        wget -c https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.tar.bz2 -O - | tar -jx -C ./

[ ! -d ./boost_1_78_0 ] && \
	wget -c https://boostorg.jfrog.io/artifactory/main/release/1.78.0/source/boost_1_78_0.tar.bz2 -O - | tar -jx -C ./


cd ${PRFX}

git clone https://github.com/kuangdai/AxiSEM-3D.git AxiSEM3D
git -C AxiSEM3D pull

mkdir -p build
cd build


rm -rf ./*

cmake -Dcxx=mpicxx \
  -Dflags="-std=c++17 -O3 -DNDEBUG -fPIC" \
  -Deigen=${PRFX}/dependencies/eigen-3.4.0 \
  -Dboost=${PRFX}/dependencies/boost_1_78_0 \
  -Dfftw=/apps/system/easybuild/software/FFTW/3.3.10-gompi-2021b \
  -Dmetis=/apps/system/easybuild/software/METIS/5.1.0-GCCcore-10.3.0 \
  -Dpar_netcdf=false \
  -Dnpol=4 \
  -Ddouble=true \
  ../AxiSEM3D/SOLVER/

make

There are a couple of system specific things to note. First, the login nodes are low-powered VMs with a different architecture to the compute nodes. This means sensible building must be done on an interactive node requested via slurm with srun -p interactive --pty /bin/bash. That just requests one core, but make -j will try and run on all cores and end up getting killed mid-build. This is why the -j flag is omitted above. Just get a coffee.

Finally, you’ll need to load modules at run time (in the slurm submission script). This works for me:

#!/bin/bash 
#SBATCH --nodes=1 
#SBATCH --ntasks-per-node=48
#SBATCH --mem-per-cpu=2G
#SBATCH --time=00:10:00 
#SBATCH --job-name=myjob 
#SBATCH --partition=devel


module load netCDF
module load FFTW

# Why does it expect input based on the path to the exe??
cp /home/eart0526/code/AxiSEM3D_april/build/axisem3d .
mpirun axisem3d 
rm axisem3d

GRACE cluster at Yale#

AxiSEM3D is installed on GRACE as a module. Simply load it via module load AxiSEM3D/2022Sep13-iomkl-2020b-avx512. To run it, just use mpirun axisem3d.