- Use an appropriate computer
- Ensure you have compilers and wrappers
- Build libraries
- Configure the
Makefile.in
in the base directory - Build
common/
andexamples/case/
- RBC3D has not been tested on WSL or Windows computers broadly. We do not recommend using WSL. Instead, use a Linux partition or a *nix-based computing cluster.
At Georgia Tech we have several, including ICE and PACE Phoenix. Automated installer scripts and run instructions for these clusters are available in install/clusters.md
.
You will need gcc
, gfortran
, and a suitable MPI wrapper like mvapich
(or the like).
-
On PACE Phoenix you can issue
module load gcc mvapich2
. -
On COC-ICE you can issue
module load gcc/12.3.0 mvapich2/2.3.7-1
-
To check for gfortran and gcc, see if
which gfortran
andwhich gcc
return a path -
Similarly, to see if you can run MPI commands for later, see if
which mpicc
orwhich mpif90
return a path
You will also need python3 for the PETSc install. On Phoenix, you can module load it via module load python/3.9.12-rkxvr6
, and ICE has a similar module. You may also need to add the ~/.local/bin
directory to your PATH by adding this line to your ~/.bashrc
:
export PATH="$PATH:$HOME/.local/bin"
- For mkl, you can
module load mkl
on the Phoenix cluster ormodule load intel-oneapi-mkl/2023.1.0
on the ICE cluster. - This will automatically set the
MKL_ROOT
environment variable necessary forMakefile.in
- You can check environment variables via
module show mkl
- If your cluster does not have mkl, it's available for download and install here.
- If MKL is not available on your system, follow the LAPACK and BLAS step, and skip this step.
- Note that
MKL_LIB
options in Makefile.in may need to be changed depending on the version of mkl, but the mkl link line advisor should provide the correct link options.
- Move back into packages:
cd RBC3D/packages
- Download the latest lapack (at time of writing
3.11
):wget https://github.com/Reference-LAPACK/lapack/archive/refs/tags/v3.11.tar.gz
- Unpack it
tar -xvf v3.11.tar.gz
cd lapack-3.11
- Modify
make.inc.example
as needed-
CC = mpicc
(line 9) -
FC = mpif90
(line 20)
-
mv make.inc.example make.inc
- Build (this takes a few minutes):
make
- Later, You will need the absolute path of
liblapack.a
andlibrefblas.a
to configurepetsc-lite
- In my case this is
/storage/coda1/p-sbryngelson3/0/sbryngelson3/RBC3D/packages/lapack-3.11/liblapack.a
- and
/storage/coda1/p-sbryngelson3/0/sbryngelson3/RBC3D/packages/lapack-3.11/librefblas.a
- In my case this is
- Note that if you're choosing to install LAPACK/BLAS instead of MKL, you'll need to replace
$(MKL_LIB) with $ (BLAS_LIB) $(LAPACK_LIB) inexamples/case/Makefile
when you run the example case later.
- This depends on MKL or LAPACK/BLAS from above, don't attempt until one of those steps is finished
- Move back to
RBC3D/packages
- Download PETSc:
wget https://ftp.mcs.anl.gov/pub/petsc/petsc-3.19.tar.gz
- Unpack
petsc-3.19
:tar -xvf petsc-3.19.tar.gz
- Descend into the directory:
cd petsc-3.19
- Run configure script. This depends on python3.
- If you installed with MKL and have the
$MKLROOT
env variable set, run this configure command:
./configure --with-cc=mpicc \
--with-cxx=mpicxx \
--with-fc=mpif90 \
--with-fortran-datatypes \
--with-debugging=0 \
--COPTFLAGS=-g -O3 -march=native -mtune=native \
--CXXOPTFLAGS=-g -O3 -march=native -mtune=native \
--FOPTFLAGS=-g -O3 -march=native -mtune=native \
--with-blaslapack-dir=$MKLROOT \
--with-mpiexec=srun \
--with-x11=0 --with-x=0 --with-windows-graphics=0
- If you installed with the LAPACK/BLAS version in an earlier step, run this configure command, and replace
$parentdir
with whatpwd
returns from the RBC3D root directory.
./configure --with-cc=mpicc \
--with-cxx=mpicxx \
--with-fc=mpif90 \
--with-fortran-datatypes \
--with-debugging=0 \
--COPTFLAGS=-g -O3 -march=native -mtune=native \
--CXXOPTFLAGS=-g -O3 -march=native -mtune=native \
--FOPTFLAGS=-g -O3 -march=native -mtune=native \
--with-blas-lib=$parentdir/packages/lapack-3.11/librefblas.a \
--with-lapack-lib=$parentdir/packages/lapack-3.11/liblapack.a \
--with-mpiexec=srun \
--with-shared-libraries=0 \
--with-x11=0 --with-x=0 --with-windows-graphics=0
- Build and Test:
make PETSC_DIR=`pwd` PETSC_ARCH=arch-linux-c-opt all
make PETSC_DIR=`pwd` PETSC_ARCH=arch-linux-c-opt check
make check
may return errors, but you can ignore these if the tests completed.
- Easiest if this is already available or can be loaded.
- PACE Phoenix has this available as a module:
module load fftw
module show fftw
tells you where the library is if you look atFFTWROOT
.- At time of writing, the FFTW library files live at
/usr/local/pace-apps/spack/packages/linux-rhel7-x86_64/gcc-10.3.0/fftw-3.3.10-dgx5szpp2x4fznqfuaoucmwieqxbgpg6/lib
- You will need this directory for the
Makefile.in
- You will need this directory for the
- If FFTW conflicts with the
gcc
ormvapich2
you module loaded, you can manually installfftw-3.3.10
in the packages directory via the command line. An example of how to do this is available ininstall/install-phoenix.sh
.CMakeLists.txt
automatically checks for this, so there's no need to set environment variables.
- Need to get to netcdff (netcdf-fortran)
- PACE Phoenix has this pre-installed
module load netcdf-c netcdf-cxx netcdf-fortran
- Later you will need information about where
netcdf-fortran
is installed forMakefile.in
. - Get this via
module show netcdf-fortran
and looking at theNETCDF_FORTRANROOT
- In my case, this is
/usr/local/pace-apps/spack/packages/linux-rhel7-x86_64/gcc-10.3.0/netcdf-fortran-4.5.4-yx5osuxluenmuvr3xnahmosfr3abeu2p/
- An example of how to install
netcdf-fortran
manually is available ininstall/install-ice
.
- Navigate back to
RBC3D/packages
- Fetch Spherepack:
git clone https://github.com/comp-physics/spherepack3.2.git
cd spherepack3.2
- Change
make.inc
if using non-GNU compilers - Execute
make
- This is needed to run
make .depend
in bothcommon/
andcase/
during build. - Sometimes available on systems by default
- PACE Phoenix doesn't seem to have it, so let's build it
- Descend into
RBC3D/packages
git clone https://github.com/comp-physics/makedepf90.git
cd makedepf90
- Modify
Makefile
, including- Line 41:
prefix
(which should be your full path to the binary build, e.g. for me:/storage/home/hcoda1/6/sbryngelson3/p-sbryngelson3-0/RBC3D/packages/makedepf90
)
- Line 41:
- Build:
make
- Install:
make install
- This will build the
makedepf90
binary in yourRBC3D/packages/makedepf90
directory
You need to change the Makefile.in
to locate all of these libraries!
This mostly just means changing the first line of Makefile.in
:
WORK_DIR = /storage/home/hcoda1/6/sbryngelson3/p-sbryngelson3-0/RBC3D
and the module directories
# Directories from loaded modules
FFTW_DIR = /usr/local/pace-apps/spack/packages/linux-rhel7-x86_64/gcc-10.3.0/fftw-3.3.10-dgx5szpp2x4fznqfuaoucmwieqxbgpg6
NETCDF_DIR = /usr/local/pace-apps/spack/packages/linux-rhel7-x86_64/gcc-10.3.0/netcdf-fortran-4.5.4-yx5osuxluenmuvr3xnahmosfr3abeu2p
- If you have an older version of
gfortran
then you will need to remove the-fallow-argument-mismatch
flag on line 39 ofMakefile.in
This is the main codebase.
- Descend into
RBC3D/common
- Execute
make .depend
- Execute
make
This is an example case.
- Descend into
RBC3D/examples/case
- Execute
make .depend
- Execute
make
In case/
you should be able to
mpiexec -n 1 ./initcond
mpiexec ./tube
or substitute in your mpi runner likesrun
.
Note that on Phoenix, srun
only works if you salloc a new node. Also, not specifying a node count via the -n
flag will use all the cores on the machine which is necessary for a simulation with high cell counts or the parallel ./initcond
file in examples/randomized_case
.
After running srun ./tube
or the equivalent, you should see x000*.dat
, wall000*.dat
, and restart files in the D
folder.
You can load the .dat
files into Paraview to visualize them.