Skip to content

Parallelize surface increment regridding #4194

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
DavidHuber-NOAA merged 37 commits into from
Nov 7, 2025
Merged

Parallelize surface increment regridding #4194

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
9afb1bf
Initial commit
DavidNew-NOAA Oct 9, 2025
694bcd0
Update ntasks_regrid
DavidNew-NOAA Oct 9, 2025
5e367cf
Debug
DavidNew-NOAA Oct 14, 2025
e2a1f84
Debug
DavidNew-NOAA Oct 14, 2025
252ea07
Merge branch 'develop' into feature/multi-regrid
DavidNew-NOAA Oct 14, 2025
b5586d7
Debug
DavidNew-NOAA Oct 14, 2025
afad1da
update
DavidNew-NOAA Oct 14, 2025
ff22f7c
Update
DavidNew-NOAA Oct 14, 2025
cc30e68
Update
DavidNew-NOAA Oct 14, 2025
c4a22a5
Debug
DavidNew-NOAA Oct 15, 2025
6d45be9
regrid updates for IAU and WCOSS2
CatherineThomas-NOAA Oct 22, 2025
e79c59e
ufs utils fix
DavidNew-NOAA Oct 23, 2025
8492559
use mpmpd to move files around
DavidNew-NOAA Oct 23, 2025
c938881
Merge branch 'develop' into feature/multi_regrid
DavidNew-NOAA Oct 23, 2025
1a8b950
Debug
DavidNew-NOAA Oct 27, 2025
15e604f
Merge branch 'develop' into feature/multi_regrid
DavidNew-NOAA Oct 27, 2025
5341621
Merge branch 'develop' into feature/multi_regrid
DavidNew-NOAA Oct 29, 2025
c261108
Shell norms
DavidNew-NOAA Oct 29, 2025
8a022dc
shellnorms
DavidNew-NOAA Oct 29, 2025
a6699aa
Merge branch 'develop' into feature/multi_regrid
DavidNew-NOAA Oct 30, 2025
99352b9
Update all env files
DavidNew-NOAA Oct 30, 2025
97c052d
Debugging
DavidNew-NOAA Oct 31, 2025
ee7d20f
Merge branch 'develop' into feature/multi_regrid
DavidNew-NOAA Oct 31, 2025
065eda9
Debug
DavidNew-NOAA Oct 31, 2025
7b4aaaf
Update ufs_utils hash
DavidNew-NOAA Nov 3, 2025
6ef24ae
Merge branch 'develop' into feature/multi_regrid
DavidNew-NOAA Nov 3, 2025
6fde359
Address reviewer comments
DavidNew-NOAA Nov 4, 2025
482503e
Add clarifying comments
DavidNew-NOAA Nov 4, 2025
ac9b700
Update ufs_utils hash to develop and add some comments
DavidNew-NOAA Nov 5, 2025
0fae84f
Merge branch 'develop' into feature/multi_regrid
DavidNew-NOAA Nov 5, 2025
6617411
UPdate
DavidNew-NOAA Nov 5, 2025
f833970
Address reviewer comments
DavidNew-NOAA Nov 6, 2025
b4287f6
Address another comment
DavidNew-NOAA Nov 6, 2025
dbff600
Missed two carrots
DavidNew-NOAA Nov 6, 2025
36c44fe
Merge branch 'develop' into feature/multi_regrid
DavidNew-NOAA Nov 6, 2025
b74707e
Take redirects out of quotes
DavidNew-NOAA Nov 6, 2025
c6bb2ee
oops
DavidNew-NOAA Nov 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions dev/parm/config/gfs/config.resources
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1317,9 +1317,15 @@ case ${step} in

"esfc")
walltime="01:15:00"
ntasks=80
ntasks=$(( NMEM_ENS * 6)) # The regrid program requires tasks to be tiles times members
threads_per_task=1
tasks_per_node=$(( max_tasks_per_node / threads_per_task ))

# Use node_numerator/node_denominator of the tasks in a node
# to leave enough memory for regridding
node_numerator=1
node_denominator=2
tasks_per_node=$(( node_numerator*max_tasks_per_node/node_denominator ))

threads_per_task_cycle=${threads_per_task}
tasks_per_node_cycle=$(( max_tasks_per_node / threads_per_task_cycle ))
;;
Expand Down
3 changes: 3 additions & 0 deletions env/AWSPW.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,9 @@ elif [[ "${step}" = "esfc" ]]; then
export NTHREADS_CYCLE=${threads_per_task_cycle:-1}
export APRUN_CYCLE="${APRUN_default} --cpus-per-task=${NTHREADS_CYCLE}"

# REGRID requires 6*NMEM_ENS tasks for reproducibility
export APRUN_REGRID="${launcher} -n ${ntasks}"

elif [[ "${step}" = "upp" ]]; then

export NTHREADS_UPP=1
Expand Down
3 changes: 3 additions & 0 deletions env/AZUREPW.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,9 @@ elif [[ "${step}" = "esfc" ]]; then
fi
export APRUN_CYCLE="${APRUN_default}"

# REGRID requires 6*NMEM_ENS tasks for reproducibility
export APRUN_REGRID="${launcher} -n ${ntasks}"

elif [[ "${step}" = "epos" ]]; then

export NTHREADS_EPOS=${NTHREADSmax}
Expand Down
5 changes: 2 additions & 3 deletions env/GAEAC5.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -285,9 +285,8 @@ case ${step} in
fi
export APRUN_CYCLE="${APRUN_default} --cpus-per-task=${NTHREADS_CYCLE}"

# REGRID requires 6 tasks for reproducibility
ntasks_regrid=6
export APRUN_REGRID="${launcher} -n ${ntasks_regrid} "
# REGRID requires 6*NMEM_ENS tasks for reproducibility
export APRUN_REGRID="${launcher} -n ${ntasks}"

;;
"epos")
Expand Down
5 changes: 2 additions & 3 deletions env/GAEAC6.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -317,9 +317,8 @@ case ${step} in
fi
export APRUN_CYCLE="${APRUN_default} --cpus-per-task=${NTHREADS_CYCLE}"

# REGRID requires 6 tasks for reproducibility
ntasks_regrid=6
export APRUN_REGRID="${launcher} -n ${ntasks_regrid} "
# REGRID requires 6*NMEM_ENS tasks for reproducibility
export APRUN_REGRID="${launcher} -n ${ntasks}"

;;
"epos")
Expand Down
3 changes: 3 additions & 0 deletions env/GOOGLEPW.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,9 @@ elif [[ "${step}" = "esfc" ]]; then
fi
export APRUN_CYCLE="${APRUN_default}"

# REGRID requires 6*NMEM_ENS tasks for reproducibility
export APRUN_REGRID="${launcher} -n ${ntasks}"

elif [[ "${step}" = "epos" ]]; then

export NTHREADS_EPOS=${NTHREADSmax}
Expand Down
5 changes: 2 additions & 3 deletions env/HERA.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -302,9 +302,8 @@ elif [[ "${step}" = "esfc" ]]; then
fi
export APRUN_CYCLE="${APRUN_default} --cpus-per-task=${NTHREADS_CYCLE}"

# REGRID requires 6 tasks for reproducibility
ntasks_regrid=6
export APRUN_REGRID="${launcher} -n ${ntasks_regrid} "
# REGRID requires 6*NMEM_ENS tasks for reproducibility
export APRUN_REGRID="${launcher} -n ${ntasks}"

elif [[ "${step}" = "epos" ]]; then

Expand Down
5 changes: 2 additions & 3 deletions env/HERCULES.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -322,9 +322,8 @@ case ${step} in
fi
export APRUN_CYCLE="${APRUN_default} --cpus-per-task=${NTHREADS_CYCLE}"

# REGRID requires 6 tasks for reproducibility
ntasks_regrid=6
export APRUN_REGRID="${launcher} -n ${ntasks_regrid} "
# REGRID requires 6*NMEM_ENS tasks for reproducibility
export APRUN_REGRID="${launcher} -n ${ntasks}"

;;
"epos")
Expand Down
5 changes: 2 additions & 3 deletions env/ORION.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -311,9 +311,8 @@ elif [[ "${step}" = "esfc" ]]; then
fi
export APRUN_CYCLE="${APRUN_default} --cpus-per-task=${NTHREADS_CYCLE}"

# REGRID requires 6 tasks for reproducibility
ntasks_regrid=6
export APRUN_REGRID="${launcher} -n ${ntasks_regrid} "
# REGRID requires 6*NMEM_ENS tasks for reproducibility
export APRUN_REGRID="${launcher} -n ${ntasks}"

elif [[ "${step}" = "epos" ]]; then

Expand Down
5 changes: 2 additions & 3 deletions env/URSA.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -291,9 +291,8 @@ elif [[ "${step}" = "esfc" ]]; then
fi
export APRUN_CYCLE="${APRUN_default} --cpus-per-task=${NTHREADS_CYCLE}"

# REGRID requires 6 tasks for reproducibility
ntasks_regrid=6
export APRUN_REGRID="${launcher} -n ${ntasks_regrid} "
# REGRID requires 6*NMEM_ENS tasks for reproducibility
export APRUN_REGRID="${launcher} -n ${ntasks}"

elif [[ "${step}" = "epos" ]]; then

Expand Down
5 changes: 2 additions & 3 deletions env/WCOSS2.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -321,9 +321,8 @@ elif [[ "${step}" = "esfc" ]]; then
fi
export APRUN_CYCLE="${APRUN_default} -ppn ${tasks_per_node_cycle} --cpu-bind depth --depth ${NTHREADS_CYCLE}"

# REGRID requires 6 tasks for reproducibility
ntasks_regrid=6
export APRUN_REGRID="${launcher} -n ${ntasks_regrid} "
# REGRID requires 6*NMEM_ENS tasks for reproducibility
export APRUN_REGRID="${launcher} -n ${ntasks}"

elif [[ "${step}" = "epos" ]]; then

Expand Down
3 changes: 3 additions & 0 deletions jobs/JGDAS_ENKF_SFC
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,9 @@ RUN=${GDUMP} YMD=${gPDY} HH=${gcyc} declare_from_tmpl -rx \
COMIN_OBS_PREV:COM_OBS_TMPL \
COMIN_ATMOS_ANALYSIS_DET_PREV:COM_ATMOS_ANALYSIS_TMPL

# Use CFP to stage and save files in parallel
export USE_CFP=YES

###############################################################
# Run relevant script

Expand Down
3 changes: 3 additions & 0 deletions jobs/JGLOBAL_ATMOS_SFCANL
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@ RUN="enkfgdas" MEMDIR="ensstat" YMD=${PDY} HH=${cyc} declare_from_tmpl -rx \

mkdir -p "${COMOUT_ATMOS_RESTART}"

# Use CFP to stage and save files in parallel
export USE_CFP=YES

###############################################################
# Run relevant script

Expand Down
2 changes: 1 addition & 1 deletion scripts/exgdas_enkf_sfc.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,7 @@ if [[ "$DOIAU" == "YES" ]]; then
cpreq "${FIXgfs}/orog/${CASE}/${CASE}_grid.tile${n}.nc" "${DATA}/fngrid.${cmem}"
cpreq "${FIXgfs}/orog/${CASE}/${CASE}.mx${OCNRES}_oro_data.tile${n}.nc" "${DATA}/fnorog.${cmem}"

if [[ "${DO_GSISOILDA}" == "YES" ]]; then
if [[ "${DO_GSISOILDA}" == "YES" ]] && [[ "${DO_LAND_IAU}" == ".false." ]]; then
cpreq "${COMIN_ATMOS_ANALYSIS_MEM}/increment.sfc.i00${LFHR}.tile${n}.nc" \
"${DATA}/soil_xainc.${cmem}"
fi
Expand Down
2 changes: 1 addition & 1 deletion sorc/ufs_utils.fd
Open in desktop
Submodule ufs_utils.fd updated 7 files
+7 −6 parm/regrid_sfc/regrid.nml_tmpl
+1 −0 reg_tests/regrid_sfc/gauss2fv3incr.sh
+23 −21 sorc/regrid_sfc.fd/grids_IO.F90
+7 −5 sorc/regrid_sfc.fd/readin_setup.F90
+45 −32 sorc/regrid_sfc.fd/regridStates.F90
+1 −0 tests/regrid_sfc/data/regrid.nml
+2 −2 tests/regrid_sfc/ftst_read_namelist.F90
Loading