Running Lethe on Compute Canada clusters

• Setting-up the folder structure
• Installing deal.ii
  • On Niagara Beluga Graham or Cedar
  • On Narval
• Installing Lethe
  • If numdiff is missing
• Copying local files
• Creating a .dealii
• Launching simulations
• Annex: saving ssh key (Linux)

Setting-up the folder structure

In your $HOME, create a "dealii" folder and a "lethe" folder, each containing "build" and "inst" folders:

mkdir -p {dealii,lethe}/{build,inst}

The deal.ii and Lethe projects can then be cloned in their corresponding folders, as indicated later in this tutorial.

After installation is complete, the folder structure will be, for deal.ii (and likewise for Lethe):

• $HOME/dealii/dealii for deal.ii git,
• $HOME/dealii/build for compilation (cmake command),
• $HOME/dealii/inst for installation (make install command)

Folders can be open with the cd command (cd $folder_path).

For the sake of clarity, this is the folder structure considered for the rest of this tutorial.

Installing deal.ii

On Niagara, Beluga, Graham or Cedar

All operations can be performed on login nodes.

⚠️ If on Niagara, this additional module is needed beforehand:

module load CCEnv

⚠️ If on Narval, see alternative installation instructions below. Note that Narval instructions use the newer Compute Canada environment and can be deployed on all clusters. They are just slightly more complicated.

For any cluster except Narval (that is: Niagara, Beluga, Graham, Cedar), load Trilinos, Parmetis and P4est, and their prerequisite modules:

module load nixpkgs/16.09
module load gcc/7.3.0
module load openmpi/3.1.2
module load trilinos/12.12.1
module load parmetis/4.0.3
module load p4est/2.0

Then, we can clone and compile dealii. Although Lethe always supports the master branch of deal.II, we maintain an identical deal.II fork on the lethe repository. This fork is always tested to make sure it works with lethe. To clone the dealii fork github repository, execute in $HOME/dealii directory:

git clone https://github.com/lethe-cfd/dealii.git

We can compile dealii in the $HOME/dealii/build folder, by defining path to installation folders of Trilinos, Parmetis and P4est:

cmake ../dealii -DDEAL_II_WITH_MPI=ON -DDEAL_II_WITH_TRILINOS=ON -DTRILINOS_DIR=$EBROOTTRILINOS -DDEAL_II_WITH_P4EST=ON -DP4EST_DIR=$EBROOTP4EST -DDEAL_II_WITH_METIS=ON -DMETIS_DIR=$EBROOTPARMETIS -DCMAKE_INSTALL_PREFIX=../inst

and :

make -j8 install

The argument -jX specifies the number of processors used for the compilation. On login nodes, a maximum of 8 cores should be used in order to ensure that other users can continue using the cluster without slowdowns.

💡 if make install triggers an error, run the command which trilinos. If it returns /usr/bin/which: no trilinos in, try removing the deal.ii build folder, recreating it and specify explicitly libraries' paths on the cmake.

On Narval

Load Trilinos, Parmetis and P4est, and their prerequisite modules:

module load gcc/9.3.0
module load openmpi/4.0.3
module load trilinos/13.0.1
module load parmetis/4.0.3
module load p4est/2.2

Then, we can clone and compile dealii. Although Lethe always supports the master branch of deal.II, we maintain an identical deal.II fork on the lethe repository. This fork is always tested to make sure it works with lethe. To clone the dealii fork github repository, execute in $HOME/dealii directory:

git clone https://github.com/lethe-cfd/dealii.git

We can compile dealii in the $HOME/dealii/build folder, by defining path to installation folders of Trilinos, Parmetis and P4est:

cmake ../dealii -DDEAL_II_WITH_MPI=ON -DDEAL_II_WITH_TRILINOS=ON -DTRILINOS_DIR=$EBROOTTRILINOS -DDEAL_II_WITH_P4EST=ON -DP4EST_DIR=$EBROOTP4EST -DDEAL_II_WITH_METIS=ON -DMETIS_DIR=$EBROOTPARMETIS -DCMAKE_INSTALL_PREFIX=../inst -DTrilinos_FIND_COMPONENTS="Pike;PikeImplicit;PikeBlackBox;TrilinosCouplings;Panzer;PanzerMiniEM;PanzerAdaptersSTK;PanzerDiscFE;PanzerDofMgr;PanzerCore;Piro;ROL;Stokhos;Tempus;Rythmos;ShyLU;ShyLU_DD;ShyLU_DDCommon;ShyLU_DDFROSch;ShyLU_DDBDDC;Zoltan2;Zoltan2Sphynx;MueLu;Moertel;NOX;Phalanx;Percept;STK;STKExprEval;STKDoc_tests;STKUnit_tests;STKBalance;STKTools;STKTransfer;STKSearchUtil;STKSearch;STKUnit_test_utils;STKNGP_TEST;STKIO;STKMesh;STKTopology;STKSimd;STKUtil;STKMath;Compadre;Intrepid2;Intrepid;Teko;FEI;Stratimikos;Ifpack2;Anasazi;Komplex;SEACAS;SEACASEx2ex1v2;SEACASTxtexo;SEACASNumbers;SEACASNemspread;SEACASNemslice;SEACASMat2exo;SEACASMapvar-kd;SEACASMapvar;SEACASMapvarlib;SEACASExplore;SEACASGrepos;SEACASGenshell;SEACASGen3D;SEACASGjoin;SEACASFastq;SEACASEx1ex2v2;SEACASExo_format;SEACASExotxt;SEACASExomatlab;SEACASExodiff;SEACASExo2mat;SEACASEpu;SEACASEjoin;SEACASConjoin;SEACASBlot;SEACASAprepro;SEACASAlgebra;SEACASPLT;SEACASSVDI;SEACASSuplibCpp;SEACASSuplibC;SEACASSuplib;SEACASSupes;SEACASAprepro_lib;SEACASChaco;SEACASIoss;SEACASNemesis;SEACASExoIIv2for32;SEACASExodus_for;SEACASExodus;Amesos2;ShyLU_Node;ShyLU_NodeTacho;ShyLU_NodeHTS;Belos;ML;Ifpack;Zoltan2Core;Pamgen;Amesos;Galeri;AztecOO;Pliris;Isorropia;Xpetra;Thyra;ThyraTpetraAdapters;ThyraEpetraExtAdapters;ThyraEpetraAdapters;ThyraCore;Domi;TrilinosSS;Tpetra;TpetraCore;TpetraTSQR;TpetraClassic;EpetraExt;Triutils;Shards;Zoltan;Epetra;MiniTensor;Sacado;RTOp;KokkosKernels;Teuchos;TeuchosKokkosComm;TeuchosKokkosCompat;TeuchosRemainder;TeuchosNumerics;TeuchosComm;TeuchosParameterList;TeuchosParser;TeuchosCore;Kokkos;KokkosAlgorithms;KokkosContainers;KokkosCore;Gtest;TrilinosATDMConfigTests;TrilinosFrameworkTests"

This command is absolutely barbaric, but it is required to prevent the usage of the Zadelus library within Trilinos 13.0.1.

and :

make -j8 install

The argument -jX specifies the number of processors used for the compilation. On login nodes, a maximum of 8 cores should be used in order to ensure that other users can continue using the cluster without slowdowns.

💡 if make install triggers an error, run the command which trilinos. If it returns /usr/bin/which: no trilinos in, try removing the deal.ii build folder, recreating it and specify explicitly libraries' paths on the cmake.

Installing Lethe

After installing deal.ii, compiling Lethe on Beluga, Niagara, Graham or Cedar is relatively straightforward, especially since all of these cluster share a very similar environment. To compile Lethe, the Trilinos, Parmetis and P4est modules should be loaded.

In the $HOME/lethe directory, download Lethe:

git clone https://github.com/lethe-cfd/lethe.git

To install Lethe in the $HOME/lethe/inst directory (applications will be in inst/bin), run in the $HOME/lethe/build directory:

cmake ../lethe  -DDEAL_II_DIR=../../dealii/inst -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=../inst
make install -j8

If numdiff is missing

If numdiff is not installed, you will have an error at the cmake step of Lethe's installation, stating that this module is missing. To install the package manually:

1. download the compressed folder (ex/ numdiff-5.9.0.tar.gz)
2. unzip it
3. copy it with scp -r to your calculcanada account on the chosen cluster (see copying local files section)
4. in the numdiff folder on the cluster, execute:

./configure
make

Finally, add it to your path environment:

PATH=$PATH:$HOME/path/to/numdiff/folder

Copying local files

On Linux, use scp (for secure copy) to copy needed files for the simulation (prm, msh):

scp /home/path/in/your/computer/*.prm [email protected]:/scratch/path/in/cluster

NB: if you need to copy a folder, use scp -r.

Simulation files must be in scratch. To get the address of your scratch folder, in your cluster account run:

cd $SCRATCH
pwd

On Windows, use third-party, such as PuTTY (see the wiki page on Transferring data))

Creating a .dealii

In order to call your dealii local installation, it is convenient to create a .dealii file in your $HOME directory:

nano .dealii

In the nano terminal, copy-paste (with Ctrl+Shift+V):

module load CCEnv #if on Niagara
module load nixpkgs/16.09
module load gcc/7.3.0
module load openmpi/3.1.2
module load p4est/2.0
module load trilinos/12.12.1
module load parmetis/4.0.3

export DEAL_II_DIR=$HOME/dealii/inst/
export PATH=$PATH:$HOME/lethe/inst/bin/

Exit the nano mode with Ctrl+x and save the document by hitting y on the prompt "Save modify buffer?" (in the bottom). The prompt "File Name to Write: .dealii" should then appear, hit Enter.

You can then source it on the terminal with:

source $HOME/.dealii

and use it in your .sh script (see Launching simulations below).

Launching simulations

Simulations are sent to the scheduler via batch scripts, visit Compute Canada wiki page more information about the scheduler and running jobs. For your convenience, an example of job.sh used on Beluga is given below:

#!/bin/bash
#SBATCH --account=$yourgroupaccount
#SBATCH --ntasks-per-node=$X #number of parallel tasks (as in mpirun -np X)
#SBATCH --nodes=1 #number of whole nodes used (each with up to 40 tasks-per-node)
#SBATCH --time=1:00:00 #maximum time for the simulation (hh:mm:ss)
#SBATCH --mem=120G #memory usage per node. See cluster specification for maximal amount.
#SBATCH --job-name=$yourjobname
#SBATCH --mail-type=END #email preferences
#SBATCH --mail-type=FAIL
#SBATCH [email protected]

source $HOME/.dealii
srun $HOME/lethe/inst/bin/$lethe_application_name_wanted $parameter_file_name.prm

The job is sent using:

sbatch job.sh

Status can be followed with the sq command: under ST, PD indicates a pending job, and R a running job.

Console outputs are written in slurm-$jobID.out. For instance, to display the 20 last lines from this file, use:

tail -n 20 slurm-$jobID.out

💡 If you need to launch multiple simulations, such as with varying parameter, feel free to adapt one of the scripts provided on lethe-utils.

Annex: saving ssh key (Linux)

To save your key on the cluster, so that it is not asked for each log or scp, generate your ssh-key with:

ssh-keygen

and copy it on the cluster:

ssh-copy-id [email protected]