Skip to content

JET update to enable fcst only run: #384

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
lgannoaa wants to merge 4 commits into from
Closed

JET update to enable fcst only run: #384

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b19ccea
JET update to enable fcst only run:
lgannoaa Jul 28, 2021
8a65b89
Include sdditional JET modulefiles and script
lgannoaa Jul 28, 2021
8791f95
Update hpc-stack and clean up the build_workflow_utils.sh script
lgannoaa Jul 28, 2021
59df4e0
Modified to remove OznMon RadMon modulefiles
lgannoaa Jul 28, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 14 additions & 13 deletions env/JET.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,19 +13,20 @@ fi

step=$1

# Theia information
export npe_node_max=24
export launcher="mpirun -np"

# Configure MPI environment
export MPI_BUFS_PER_PROC=2048
export MPI_BUFS_PER_HOST=2048
export MPI_GROUP_MAX=256
export MPI_MEMMAP_OFF=1
export MP_STDOUTMODE="ORDERED"
if [[ "$PARTITION_BATCH" = "xjet" ]]; then
export npe_node_max=24
elif [[ "$PARTITION_BATCH" = "vjet" ]]; then
export npe_node_max=16
fi
export launcher="srun --export=ALL"

# Configure STACK
export OMP_STACKSIZE=2048000
export NTHSTACK=1024000000
#export LD_BIND_NOW=1

ulimit -s unlimited
ulimit -a

export job=${PBS_JOBNAME:-$step}
export jobid=${job}.${PBS_JOBID:-$$}

Expand All @@ -34,8 +35,8 @@ if [ $step = "prep" -o $step = "prepbufr" ]; then
nth_max=$(($npe_node_max / $npe_node_prep))

export POE="NO"
export BACK=${BACK:-"YES"}
export sys_tp="Cray-CS400"
export BACK=${BACK:-"NO"}
export sys_tp="JET"

elif [ $step = "waveinit" -o $step = "waveprep" -o $step = "wavepostsbs" -o $step = "wavepostbndpnt" -o $step = "wavepostbndpntbll" -o $step = "wavepostpnt" ]; then

Expand Down
45 changes: 20 additions & 25 deletions modulefiles/module_base.jet
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,32 +1,27 @@
#%Module######################################################################
##
## nems prerequisites
## FV3GFS prerequisites
##

proc ModulesHelp {} {
puts stderr "The prerequisites for compiling or running FV3 on Jet. "
}
# Load hpc-stack
module use /lfs4/HFIP/hfv3gfs/nwprod/hpc-stack/libs/modulefiles/stack
module load hpc/1.1.0

module load newdefaults intel/15.0.3.187 impi/5.1.3.181 szip hdf5 netcdf4/4.2.1.1
# Load intel compiler and mpi
module load hpc-intel/18.0.5.274
module load hpc-impi/2018.4.274

module use /lfs3/projects/hfv3gfs/nwprod/lib/modulefiles
module load bacio/v2.0.2
module load sp/v2.0.2
module load ip/v2.0.0
module load w3nco/v2.0.6
module load w3emc/v2.2.0
module load nemsio/v2.2.2

#set NCEPLIBS $::env(NCEPLIBS)
#module use $NCEPLIBS/modulefiles
#module load esmf/7.1.0r_impi

module use /lfs3/projects/hwrf-vd/soft/modulefiles
module load prod_util
module load grib_util
module load wgrib2/2.0.8
module load hpss

# mpiserial on Jet includes a cfp wrapper
module load mpiserial

module load nco/4.9.1
module load cdo/1.9.5
module load gempak/7.4.2

module load prod_util/v1.0.18
module load grib_util/v1.1.1
module load g2tmpl/1.6.0
module load crtm/2.3.0
module load esmflocal/8_0_1
module load netcdfp/4.7.4
Comment on lines +16 to +24
Copy link
Copy Markdown
Contributor

@aerorahul aerorahul Jul 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please make sure these are built with hpc-stack.
I know for a fact hpc-stack does not build netcdfp or esmflocal modules for these

Copy link
Copy Markdown
Contributor Author

@lgannoaa lgannoaa Jul 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am testing this item on Jet now.

Copy link
Copy Markdown
Contributor

@aerorahul aerorahul Jul 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some of these are still not from hpc-stack. So if you are testing with these modules, then they are likely wrong.


# rocoto
module load rocoto
32 changes: 32 additions & 0 deletions modulefiles/workflow_utils.jet
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
#%Module#####################################################
## Workflow Utilities - hera
#############################################################

# Load cmake
module load cmake/3.16.1

# Load hpc-stack
module use /lfs4/HFIP/hfv3gfs/nwprod/hpc-stack/libs/modulefiles/stack
module load hpc/1.1.0

# Load intel compiler and mpi
module load hpc-intel/18.0.5.274
module load hpc-impi/2018.4.274

module load jasper/2.0.25
module load zlib/1.2.11
module load png/1.6.35

module load bacio/2.4.1
module load w3nco/2.4.1
module load w3emc/2.7.3
module load sp/2.3.3
module load ip/3.3.3
module load nemsio/2.5.2
module load nemsiogfs/2.5.3
module load sigio/2.3.2
module load g2/3.4.1
module load bufr/11.4.0

module load hdf5/1.10.6
module load netcdf/4.7.4
Comment on lines +31 to +32
Copy link
Copy Markdown
Contributor

@aerorahul aerorahul Jul 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you need to load these after png.
netcdf is used in w3 library.

Copy link
Copy Markdown
Contributor Author

@lgannoaa lgannoaa Jul 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am testing this item on Jet now.

1 change: 1 addition & 0 deletions sorc/build_fv3.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ fi

if [ $target = hera ]; then target=hera.intel ; fi
if [ $target = orion ]; then target=orion.intel ; fi
if [ $target = jet ]; then target=jet.intel ; fi

cd fv3gfs.fd/
FV3=$( pwd -P )/FV3
Expand Down
6 changes: 6 additions & 0 deletions sorc/build_gfs_util.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,12 @@ if [ ! -d "./mkgfsawps.fd" ]; then
exit
fi

if [ $target = jet ]; then
echo " GFS_UTIL does not support JET "
echo " "
exit
fi

Comment on lines +17 to +22
Copy link
Copy Markdown
Contributor

@aerorahul aerorahul Jul 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't need this block. please remove.
We don't check for every machine here, so no need to do check for jet here.

echo ""
echo " Building ... Executables for GFS_UTILITIES "
echo ""
Expand Down
6 changes: 6 additions & 0 deletions sorc/build_gldas.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ set -eux
source ./machine-setup.sh > /dev/null 2>&1
cwd=`pwd`

if [ $target = jet ]; then
echo " GLDAS does not support JET "
echo " "
exit
fi

Comment on lines +7 to +12
Copy link
Copy Markdown
Contributor

@aerorahul aerorahul Jul 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove.

USE_PREINST_LIBS=${USE_PREINST_LIBS:-"true"}
if [ $USE_PREINST_LIBS = true ]; then
export MOD_PATH=/scratch3/NCEPDEV/nwprod/lib/modulefiles
Expand Down
7 changes: 7 additions & 0 deletions sorc/build_ww3prepost.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,13 @@ finalexecdir=$( pwd -P )/../exec

set +x
source ./machine-setup.sh > /dev/null 2>&1

if [ $target = jet ]; then
echo " WW3 does not support JET "
echo " "
exit
fi

Comment on lines +13 to +19
Copy link
Copy Markdown
Contributor

@aerorahul aerorahul Jul 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please remove

source ../modulefiles/modulefile.ww3.$target
set -x

Expand Down
10 changes: 6 additions & 4 deletions sorc/link_fv3gfs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,16 @@ machine=${2}

if [ $# -lt 2 ]; then
echo '***ERROR*** must specify two arguements: (1) RUN_ENVIR, (2) machine'
echo ' Syntax: link_fv3gfs.sh ( nco | emc ) ( cray | dell | hera | orion )'
echo ' Syntax: link_fv3gfs.sh ( nco | emc ) ( cray | dell | hera | orion | jet )'
exit 1
fi

if [ $RUN_ENVIR != emc -a $RUN_ENVIR != nco ]; then
echo 'Syntax: link_fv3gfs.sh ( nco | emc ) ( cray | dell | hera | orion )'
echo 'Syntax: link_fv3gfs.sh ( nco | emc ) ( cray | dell | hera | orion | jet )'
exit 1
fi
if [ $machine != cray -a $machine != dell -a $machine != hera -a $machine != orion ]; then
echo 'Syntax: link_fv3gfs.sh ( nco | emc ) ( cray | dell | hera | orion )'
if [ $machine != cray -a $machine != dell -a $machine != hera -a $machine != orion -a $machine != jet ]; then
echo 'Syntax: link_fv3gfs.sh ( nco | emc ) ( cray | dell | hera | orion | jet )'
exit 1
fi

Expand All @@ -38,6 +38,8 @@ elif [ $machine = "hera" ]; then
FIX_DIR="/scratch1/NCEPDEV/global/glopara/fix"
elif [ $machine = "orion" ]; then
FIX_DIR="/work/noaa/global/glopara/fix"
elif [ $machine = "jet" ]; then
FIX_DIR="/lfs4/HFIP/hfv3gfs/glopara/git/fv3gfs/fix"
fi
cd ${pwd}/../fix ||exit 8
dirs=`ls $FIX_DIR`
Expand Down
Loading