Skip to content

Modify Met Office regtest matrix to run regtests on the new EX machine #1448

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JessicaMeixner-NOAA merged 2 commits into from
Jun 9, 2025
Merged

Modify Met Office regtest matrix to run regtests on the new EX machine #1448

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a25bc96
Modify Met Office regtest matrix to run regtests on the new EX machine
ukmo-juan-castillo Jun 3, 2025
092a71e
Update ukmo regtest matrix to point to the newly compiled parmetis and
ukmo-juan-castillo Jun 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 19 additions & 21 deletions regtests/bin/matrix_cmake_ukmo_cray
View file
Open in desktop
Copy link
Copy Markdown
Collaborator

@ukmo-ccbunney ukmo-ccbunney Jun 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One other thing that I had to add to the cmake build were the following lines:

# For CMAKE
echo "export CC=cc"             >> matrix.head
echo "export FTN=ftn"           >> matrix.head

This was because cmake would quietly go and find the crayftn compiler rather than using the proper wrapper scripts. This caused me linker problems originally as the ftn wrappers script sets the library paths.

Copy link
Copy Markdown
Collaborator Author

@ukmo-juan-castillo ukmo-juan-castillo Jun 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, I will do so. Is there any documentation on how to compile ParMETIS? I have some test failing and I think it is because I just copied the files from the XC40s, and there are library versions incompatibility...

Copy link
Copy Markdown
Collaborator

@ukmo-ccbunney ukmo-ccbunney Jun 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might want to focus on getting SCOTCH compiled instead? IIRC ParMETIS was going to be replaced by SCOTCH.
WE could not compile scotch on our old HPC due to the libraries being too old. I think it should be fine on the new HPC though. I'll ping you some instructions regarding ParMETIS.

Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ EOF
echo '#PBS -l ncpus=16' >> matrix.head
echo '#PBS -l mem=16GB' >> matrix.head
echo '#PBS -q shared' >> matrix.head
echo '#PBS -l walltime=04:00:00' >> matrix.head
echo '#PBS -l walltime=03:00:00' >> matrix.head
echo '#PBS -N ww3_regtest' >> matrix.head
echo '#PBS -j oe' >> matrix.head
echo '#PBS -o matrix.out' >> matrix.head
Expand All @@ -78,38 +78,36 @@ EOF
if [[ $cmplr == "ukmo_cray" ]] || [[ $cmplr == "ukmo_cray_debug" ]]; then
# Load targetted versions of Cray Development Tools (bug in Fortran StreamIO
# for older versions) and netCDF/HDF5 modules:
echo "module load cdt/18.12" >> matrix.head
echo "module load cray-netcdf/4.6.1.3" >> matrix.head
echo "module load cray-hdf5/1.10.2.0" >> matrix.head
echo "export METIS_PATH=/home/d02/frey/WW3/ParMETIS" >> matrix.head
echo "module switch PrgEnv-cray PrgEnv-cray/8.4.0" >> matrix.head
echo "module load cpe/23.05" >> matrix.head
echo "module switch cce cce/15.0.0" >> matrix.head
echo "module load cray-hdf5-parallel/1.12.2.1" >> matrix.head
echo "module load cray-netcdf-hdf5parallel/4.9.0.1" >> matrix.head
Comment on lines +84 to +85
Copy link
Copy Markdown
Collaborator

@ukmo-ccbunney ukmo-ccbunney Jun 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my original tests, I didn't use the parallel netcdf/hdf5 versions (they're not required for the netcdf pre/post-processors). I was using these non-parallel versions (which are slightly newer versions):

module load cray-hdf5/1.12.2.3
module load cray-netcdf/4.9.0.3

Perhaps it's fine to use the parallel versions? But it seems unnecessary. I'll leave that decision up to you.

Copy link
Copy Markdown
Collaborator Author

@ukmo-juan-castillo ukmo-juan-castillo Jun 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it does not matter much, I just added the default libraries for running global and regional coupled configurations at the Met Office.

echo "export METIS_PATH=/data/users/juan.m.castillo/REGTESTS/parmetis-4.0.3_cce" >> matrix.head
echo "export SCOTCH_PATH=/data/users/juan.m.castillo/REGTESTS/scotch_cce" >> matrix.head

elif [[ $cmplr == ukmo_cray_gnu* ]]; then
# ParMETIS library not currently working with Cray compiler.
# Use GNU compiler for programs that use PDLIB.
echo "module switch PrgEnv-cray PrgEnv-gnu/5.2.82" >> matrix.head
echo "module load cray-netcdf" >> matrix.head
echo "export METIS_PATH=/home/d02/frey/WW3/ParMETIS_GNU" >> matrix.head

elif [[ $cmplr == ukmo_cray_intel* ]]; then
echo "module switch PrgEnv-cray PrgEnv-intel" >> matrix.head
echo "module swap intel/15.0.0.090 intel/18.0.5.274" >> matrix.head
echo "module load cdt/18.12" >> matrix.head
echo "module load cray-netcdf/4.6.1.3" >> matrix.head
echo "module load cray-hdf5/1.10.2.0" >> matrix.head
echo "module switch PrgEnv-cray PrgEnv-gnu/8.4.0" >> matrix.head
echo "module load cpe/23.05" >> matrix.head
echo "module load cray-hdf5-parallel/1.12.2.1" >> matrix.head
echo "module load cray-netcdf-hdf5parallel/4.9.0.1" >> matrix.head
echo "export METIS_PATH=/data/users/juan.m.castillo/REGTESTS/parmetis-4.0.3_gnu" >> matrix.head
echo "export SCOTCH_PATH=/data/users/juan.m.castillo/REGTESTS/scotch_gnu" >> matrix.head

else
echo "Unknown compiler for UKMO regression tests: $cmplr"
exit 1
fi

# Need newer cmake version on the XC
echo "module load cmake/3.21.3" >> matrix.head

# SNP Launcher 7.7.4 allows -np switch:
echo "module load cray-snplauncher/7.7.4" >> matrix.head
echo "export NETCDF_CONFIG=\$(which nc-config)" >> matrix.head

# On the Cray XC, we need to stop CMake from searching for
# For CMAKE
echo "export CC=cc" >> matrix.head
echo "export FTN=ftn" >> matrix.head

# On the EX, we need to stop CMake from searching for
# the NetCDF libraries - they are provided by the ftn wrapper.
echo "export CMAKE_OPTIONS=-DEXCLUDE_FIND=\"netcdf\"" >> matrix.head

Expand Down
36 changes: 18 additions & 18 deletions regtests/bin/matrix_ukmo_cray
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@
echo '#PBS -l ncpus=16' >> matrix.head
echo '#PBS -l mem=16GB' >> matrix.head
echo '#PBS -q shared' >> matrix.head
echo '#PBS -l walltime=04:00:00' >> matrix.head
echo '#PBS -l walltime=03:00:00' >> matrix.head
echo '#PBS -N ww3_regtest' >> matrix.head
echo '#PBS -j oe' >> matrix.head
echo '#PBS -o matrix.out' >> matrix.head
Expand All @@ -63,34 +63,34 @@
if [[ $cmplr == "ukmo_cray" ]] || [[ $cmplr == "ukmo_cray_debug" ]]; then
# Load targetted versions of Cray Development Tools (bug in Fortran StreamIO
# for older versions) and netCDF/HDF5 modules:
echo "module load cdt/18.12" >> matrix.head
echo "module load cray-netcdf/4.6.1.3" >> matrix.head
echo "module load cray-hdf5/1.10.2.0" >> matrix.head
echo "export METIS_PATH=/home/d02/frey/WW3/ParMETIS" >> matrix.head
echo "module switch PrgEnv-cray PrgEnv-cray/8.4.0" >> matrix.head
echo "module load cpe/23.05" >> matrix.head
echo "module switch cce cce/15.0.0" >> matrix.head
echo "module load cray-hdf5-parallel/1.12.2.1" >> matrix.head
echo "module load cray-netcdf-hdf5parallel/4.9.0.1" >> matrix.head
echo "export METIS_PATH=/data/users/juan.m.castillo/REGTESTS/parmetis-4.0.3_cce" >> matrix.head
echo "export SCOTCH_PATH=/data/users/juan.m.castillo/REGTESTS/scotch_cce" >> matrix.head

elif [[ $cmplr == ukmo_cray_gnu* ]]; then
# ParMETIS library not currently working with Cray compiler.
# Use GNU compiler for programs that use PDLIB.
echo "module switch PrgEnv-cray PrgEnv-gnu/5.2.82" >> matrix.head
echo "module load cray-netcdf" >> matrix.head
echo "export METIS_PATH=/home/d02/frey/WW3/ParMETIS_GNU" >> matrix.head

elif [[ $cmplr == ukmo_cray_intel* ]]; then
echo "module switch PrgEnv-cray PrgEnv-intel" >> matrix.head
echo "module swap intel/15.0.0.090 intel/18.0.5.274" >> matrix.head
echo "module load cdt/18.12" >> matrix.head
echo "module load cray-netcdf/4.6.1.3" >> matrix.head
echo "module load cray-hdf5/1.10.2.0" >> matrix.head
echo "module switch PrgEnv-cray PrgEnv-gnu/8.4.0" >> matrix.head
echo "module load cpe/23.05" >> matrix.head
echo "module load cray-hdf5-parallel/1.12.2.1" >> matrix.head
echo "module load cray-netcdf-hdf5parallel/4.9.0.1" >> matrix.head
echo "export METIS_PATH=/data/users/juan.m.castillo/REGTESTS/parmetis-4.0.3_gnu" >> matrix.head
echo "export SCOTCH_PATH=/data/users/juan.m.castillo/REGTESTS/scotch_gnu" >> matrix.head

else
echo "Unknown compiler for UKMO regression tests: $cmplr"
exit 1
fi

# SNP Launcher 7.7.4 allows -np switch:
echo " module load cray-snplauncher/7.7.4" >> matrix.head
echo " export NETCDF_CONFIG=\$(which nc-config)" >> matrix.head

echo " export NETCDF_CONFIG=\$(which nc-config)" >> matrix.head
# For CMAKE
echo "export CC=cc" >> matrix.head
echo "export FTN=ftn" >> matrix.head

# Compiler option. Set cmplOption to
# y if using for the first time or using a different compiler
Expand Down
Loading