Skip to content

[develop]: Update for Gaea software stack location and Lmod initialization script #627

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
MichaelLueken merged 4 commits into from
Mar 10, 2023
Merged

[develop]: Update for Gaea software stack location and Lmod initialization script #627

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9aa36ff
Updating Gaea hpc-stack location to match UFS-WM
Feb 23, 2023
a1e4d9b
Update for Gaea Lmod initialization script
Feb 23, 2023
a45a9e1
Update lmod-setup.csh, use Lmod_init.csh initialization script
natalie-perlin Feb 27, 2023
1dc3174
Update lmod-setup.csh, call .../lmod/lmod/init/csh
natalie-perlin Mar 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 1 addition & 4 deletions etc/lmod-setup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,11 +43,8 @@ elif [ "$L_MACHINE" = singularity ]; then
module purge

elif [ "$L_MACHINE" = gaea ]; then
export BASH_ENV="/lustre/f2/dev/role.epic/contrib/apps/lmod/lmod/init/bash"
source $BASH_ENV
source /lustre/f2/dev/role.epic/contrib/Lmod_init.sh

@danielabdi-noaa danielabdi-noaa Feb 23, 2023

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is best to call the "bash" script directly instead of what looks like a custom "Lmod_init.sh" script.
Could you please print the contents of the script so that we can see what it does differently?

@natalie-perlin natalie-perlin Feb 23, 2023

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The initialization script for Lmod 8.7.12 stores a list of modules that are loaded by default into the user environment, purges the modules, sources the /lustre/f2/dev/role.epic/contrib/apps/lmod/lmod/init/profile, and then loads the default modules in a preferred order, due to some interdependencies. After a default module management manager on Gaea, modules/3.2.11.4 , is loaded, $MODULEHOME is changed to point to a path set by modules/3.2.11.4. In the end of the updated Lmod initialization script, $MODULEHOME is reset to correspond to Lmod 8.7.12 module management package.

@natalie-perlin natalie-perlin Feb 23, 2023 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue is that the modules/3.2.11.4 is always a default module, but the list of the rest of the modules differ depending on whether you got into a login node or into a compute node during the model run.

@natalie-perlin natalie-perlin Feb 23, 2023

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Lmod_init.sh has the following:

#!/bin/bash
  
loaded_modules=$(echo ${LOADEDMODULES:-} | tr ":" "\n")
module purge 2>/dev/null

echo "Initializing lua module environment Lmod 8.7.12, loading modules (wait...)"
export LMOD_SYSTEM_DEFAULT_MODULES=modules/3.2.11.4
export BASH_ENV=/lustre/f2/dev/role.epic/contrib/apps/lmod/lmod/init/profile
source $BASH_ENV
export PATH=$MODULESHOME/libexec:$MODULESHOME/init/ksh_funcs:$PATH
module --initial_load --no_redirect restore
#
if [[ -d /opt/cray/ari/modulefiles ]] ; then
    module use -a /opt/cray/ari/modulefiles
fi
if [[ -d /opt/cray/pe/ari/modulefiles ]] ; then
    module use -a /opt/cray/pe/ari/modulefiles
fi
if [[ -d /opt/cray/pe/craype/default/modulefiles ]] ; then
    module use -a /opt/cray/pe/craype/default/modulefiles
fi
# Load craype module first, then DefApps, then all others
for module in $loaded_modules
do
        [[ $module == craype/* ]] &&  module try-load $module
done
for module in $loaded_modules
do
        [[ $module == DefApps ]] &&  module try-load $module
done
for module in $loaded_modules
do
        [[ $module == craype/* || $module == DefApps ]] ||  module is-loaded $module || module try-load  $module
done

# Set environment variables
export MODULESHOME=$LMOD_ROOT/lmod
#
# Report when done loading
#echo "... done loading "

@danielabdi-noaa danielabdi-noaa Feb 23, 2023

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@natalie-perlin Thanks for the details. I was not aware the logic for replacing cray modules has grown now, makes sense to put it in its own script.


export LMOD_SYSTEM_DEFAULT_MODULES="modules/3.2.11.4"
module --initial_load --no_redirect restore
Comment thread
danielabdi-noaa marked this conversation as resolved.
elif [ "$L_MACHINE" = odin ]; then
module unload modules
unset -f module
Expand Down
2 changes: 1 addition & 1 deletion modulefiles/build_gaea_intel.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ whatis([===[Loads libraries needed for building the UFS SRW App on Gaea ]===])

load(pathJoin("cmake", os.getenv("cmake_ver") or "3.20.1"))

prepend_path("MODULEPATH","/lustre/f2/dev/role.epic/contrib/hpc-stack/intel-2021.3.0/modulefiles/stack")
prepend_path("MODULEPATH","/lustre/f2/dev/role.epic/contrib/hpc-stack/intel-2021.3.0_noarch/modulefiles/stack")
load(pathJoin("hpc", os.getenv("hpc_ver") or "1.2.0"))
load(pathJoin("intel", os.getenv("intel_ver") or "2021.3.0"))
load(pathJoin("hpc-intel", os.getenv("hpc_intel_ver") or "2021.3.0"))
Expand Down