Parallelize surface increment regridding by DavidNew-NOAA · Pull Request #1109 · ufs-community/UFS_UTILS

DavidNew-NOAA · 2025-10-14T19:22:00Z

DESCRIPTION OF CHANGES:

This PR, a companion to NOAA-EMC/global-workflow#4193, parallelizes the surface increment regridding code over multiple ensemble members. Now, one can specify the number of ensemble members and specify multiple input directories (one for each ensemble member) in the namelist.

TESTS CONDUCTED:

If there are changes to the build or source code, the tests below must be conducted. Contact a repository manager if you need assistance.

Optional test.

Run full set of chgres_cube consistency tests on Ursa.

Describe any additional tests performed.

DEPENDENCIES:

None

DOCUMENTATION:

All new and updated source code must be documented with Doxygen.

Doxygen is updated.

BrianCurtis-NOAA · 2025-10-30T15:27:20Z

@DavidNew-NOAA could you also make corresponding changes in https://github.com/DavidNew-NOAA/UFS_UTILS/blob/feature/multi_regrid/tests/regrid_sfc/data/regrid.nml to match ?

BrianCurtis-NOAA · 2025-10-30T17:16:06Z

@DavidNew-NOAA could you also make corresponding changes in https://github.com/DavidNew-NOAA/UFS_UTILS/blob/feature/multi_regrid/tests/regrid_sfc/data/regrid.nml to match ?

Just posting FYI. Here is the unit test failure:

57/57 Testing: regrid_sfc-ftst_read_namelist
57/57 Test: regrid_sfc-ftst_read_namelist
Command: "/scratch4/NCEPDEV/nems/Brian.Curtis/git/DavidNew-NOAA/UFS_UTILS/build/tests/regrid_sfc/ftst_read_namelist"
Directory: /scratch4/NCEPDEV/nems/Brian.Curtis/git/DavidNew-NOAA/UFS_UTILS/build/tests/regrid_sfc
"regrid_sfc-ftst_read_namelist" start time: Oct 30 15:23 UTC
Output:
----------------------------------------------------------
 Starting test of readin_setup.
 Read input namelist.
 - FATAL ERROR: unknown namel in readin_setup
 - IOSTAT IS:            1
Attempting to use an MPI routine (PMPI_Abort) before initializing or after finalizing MPICH

BrianCurtis-NOAA · 2025-10-30T17:28:38Z

The following is from running the driver script for the regrid_sfc reg-tests:

+ srun /scratch4/NCEPDEV/nems/Brian.Curtis/git/DavidNew-NOAA/UFS_UTILS/reg_tests/regrid_sfc/../../exec/regridStates.x
 ** pets: local, total:            1           6
 ** pets: local, total:            2           6
 ** pets: local, total:            5           6
 ** pets: local, total:            0           6
 ** pets: local, total:            3           6
 ** pets: local, total:            4           6
 - FATAL ERROR:
 number of processor divided by number of tiles must equal number of ensemble me
 mbers
 - IOSTAT IS:            1
 - FATAL ERROR:
Abort(999) on node 1 (rank 1 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 999) - process 1
 number of processor divided by number of tiles must equal number of ensemble me
 mbers
 - IOSTAT IS:            1
Abort(999) on node 2 (rank 2 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 999) - process 2
 - FATAL ERROR:
 number of processor divided by number of tiles must equal number of ensemble me
 mbers
 - IOSTAT IS:            1
Abort(999) on node 3 (rank 3 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 999) - process 3
 - FATAL ERROR:
 number of processor divided by number of tiles must equal number of ensemble me
 mbers
 - IOSTAT IS:            1
Abort(999) on node 4 (rank 4 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 999) - process 4
 - FATAL ERROR:
 number of processor divided by number of tiles must equal number of ensemble me
 mbers
 - IOSTAT IS:            1
Abort(999) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 999) - process 0
 - FATAL ERROR:

DavidNew-NOAA · 2025-10-30T17:30:23Z

Thanks @BrianCurtis-NOAA . I will work on getting that unit test fixed

DavidNew-NOAA · 2025-10-31T16:00:59Z

Hi @BrianCurtis-NOAA I think I made the necessary change to the unit test namelist

BrianCurtis-NOAA · 2025-10-31T17:15:43Z

Hi @BrianCurtis-NOAA I think I made the necessary change to the unit test namelist

Still not working. Neither are my regression tests.

+ srun /scratch4/NCEPDEV/nems/Brian.Curtis/git/DavidNew-NOAA/UFS_UTILS/reg_tests/regrid_sfc/../../exec/regridStates.x
 ** pets: local, total:            1           6
 ** pets: local, total:            2           6
 ** pets: local, total:            5           6
 ** pets: local, total:            0           6
 ** pets: local, total:            3           6
 ** pets: local, total:            4           6
 - FATAL ERROR:
 number of processor divided by number of tiles must equal number of ensemble me
 mbers
 - IOSTAT IS:            1
 - FATAL ERROR:
Abort(999) on node 1 (rank 1 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 999) - process 1
 number of processor divided by number of tiles must equal number of ensemble me
 mbers
 - IOSTAT IS:            1
Abort(999) on node 2 (rank 2 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 999) - process 2
 - FATAL ERROR:
 number of processor divided by number of tiles must equal number of ensemble me
 mbers
 - IOSTAT IS:            1
Abort(999) on node 3 (rank 3 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 999) - process 3
 - FATAL ERROR:
 number of processor divided by number of tiles must equal number of ensemble me
 mbers
 - IOSTAT IS:            1
Abort(999) on node 4 (rank 4 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 999) - process 4
 - FATAL ERROR:
 number of processor divided by number of tiles must equal number of ensemble me
 mbers
 - IOSTAT IS:            1
Abort(999) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 999) - process 0
 - FATAL ERROR:
 number of processor divided by number of tiles must equal number of ensemble me
 mbers
 - IOSTAT IS:            1
Abort(999) on node 5 (rank 5 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 999) - process 5

DavidNew-NOAA · 2025-10-31T17:20:45Z

Thanks @BrianCurtis-NOAA . Are these regression tests easy to run? Can you provide instructions to get me started? I really need to play around with this myself to get a handle on what's going wrong.

BrianCurtis-NOAA · 2025-10-31T17:26:22Z

Thanks @BrianCurtis-NOAA . Are these regression tests easy to run? Can you provide instructions to get me started? I really need to play around with this myself to get a handle on what's going wrong.

cd reg-tests/regrid_sfc
edit the driver.sh script for the system you're on and your dir setup (mainly edit the WORK_DIR and PROJECT_CODE, leave HOMEreg alone)
make sure you run the link_fixdirs.sh in fix dir and built using build_all.sh
run driver.sh
look for consistency.log01

DavidNew-NOAA · 2025-10-31T17:27:30Z

@BrianCurtis-NOAA Awesome, thanks. I'll take a look this afternoon

DavidNew-NOAA · 2025-11-03T17:27:15Z

@BrianCurtis-NOAA I'm running into the following error upon running driver.sh. Is there an obvious solution to this?:

/scratch3/NCEPDEV/da/David.New/global-workflow-parallel/sorc/ufs_utils.fd/reg_tests/regrid_sfc/../../exec/regridStates.x: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /scratch3/NCEPDEV/da/David.New/global-workflow-parallel/sorc/ufs_utils.fd/reg_tests/regrid_sfc/../../exec/regridStates.x)

BrianCurtis-NOAA · 2025-11-03T17:44:57Z

@BrianCurtis-NOAA I'm running into the following error upon running driver.sh. Is there an obvious solution to this?:

/scratch3/NCEPDEV/da/David.New/global-workflow-parallel/sorc/ufs_utils.fd/reg_tests/regrid_sfc/../../exec/regridStates.x: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /scratch3/NCEPDEV/da/David.New/global-workflow-parallel/sorc/ufs_utils.fd/reg_tests/regrid_sfc/../../exec/regridStates.x)

It's got to do with the build/linking for that executable. There could be many ways it's broken.

I would try building again with intelllvm using the build_all.sh script, maybe try removing the build dir for a fresh build from src.

If it still persists, try checking what you have loaded in lua on a fresh terminal session to see if it's getting in the way of what's shown in modulefiles/build.<machine>.<compiler>.lua

DavidNew-NOAA · 2025-11-03T17:55:09Z

@BrianCurtis-NOAA Dumb mistake on my part. I built Global Workflow on Hera and was trying to run reg tests on Ursa because driver.sh didn't have settings for Hera. I rebuilt on Ursa and it works now

BrianCurtis-NOAA · 2025-11-03T17:56:12Z

@BrianCurtis-NOAA Dumb mistake on my part. I built Global Workflow on Hera and was trying to run reg tests on Ursa because driver.sh didn't have settings for Hera. I rebuilt on Ursa and it works now

Been there, done that. 👍

DavidNew-NOAA · 2025-11-03T18:20:46Z

@BrianCurtis-NOAA That last commit did the trick

BrianCurtis-NOAA · 2025-11-03T18:37:15Z

@BrianCurtis-NOAA That last commit did the trick

Reg tests ran successfully! One last thing, in tests/regrid_sfc/ftst_read_namelist.F90 can you edit it (for both grid_setup_in and grid_setup_out) to match your changes to the nml.

DavidNew-NOAA · 2025-11-03T18:50:05Z

@BrianCurtis-NOAA It looks like tests/regrid_sfc/ftst_read_namelist.F90 is only checking values for the &input and &output blocks for the namelist, and the only change I made to the namelist structure is in the &config block (nmem_ens). The other changes to parm/regrid_sfc/regrid.nml_tmpl are to the templating and not the namelist structure itself

BrianCurtis-NOAA · 2025-11-03T18:59:56Z

@BrianCurtis-NOAA It looks like tests/regrid_sfc/ftst_read_namelist.F90 is only checking values for the &input and &output blocks for the namelist, and the only change I made to the namelist structure is in the &config block (nmem_ens). The other changes to parm/regrid_sfc/regrid.nml_tmpl are to the templating and not the namelist structure itself

OK, I see. Maybe it's with the readin_setup.F90 in sorc/regrid_sfc.fd, the unit test errors out with "unknown namel in readin_setup" (line 69) . There's something in there that got messed up with the nml changes, possibly.

BrianCurtis-NOAA · 2025-11-03T19:15:40Z

@BrianCurtis-NOAA It looks like tests/regrid_sfc/ftst_read_namelist.F90 is only checking values for the &input and &output blocks for the namelist, and the only change I made to the namelist structure is in the &config block (nmem_ens). The other changes to parm/regrid_sfc/regrid.nml_tmpl are to the templating and not the namelist structure itself

OK, I see. Maybe it's with the readin_setup.F90 in sorc/regrid_sfc.fd, the unit test errors out with "unknown namel in readin_setup" (line 69) . There's something in there that got messed up with the nml changes, possibly.

I think i see it. In the ftest, the function call for readin_setup has added variables which are not present in the test.

DavidNew-NOAA · 2025-11-03T19:25:32Z

Ok, will take a look shortly

DavidNew-NOAA · 2025-11-03T20:47:39Z

@BrianCurtis-NOAA Is the unit test different from the regression test? How do you run the unit test?

BrianCurtis-NOAA · 2025-11-03T20:53:17Z

@BrianCurtis-NOAA Is the unit test different from the regression test? How do you run the unit test?

it is, yes.

(edit CMakeLists.txt and uncomment ctest (near the end))
rm -rf build
export BUILD_TESTING=ON && ./build_all.sh

This should run all unit test, they go quick and all but the last one should pass.

DavidNew-NOAA · 2025-11-03T21:52:57Z

@BrianCurtis-NOAA CTest fixed

CatherineThomas-NOAA · 2025-11-04T16:41:44Z

Thanks for your help @BrianCurtis-NOAA. Do you have any other concerns with this PR?

FYI we are going to include this PR in the GFSv17 retrospectives, which will be starting as soon as possible.

BrianCurtis-NOAA · 2025-11-04T16:43:17Z

Thanks for your help @BrianCurtis-NOAA. Do you have any other concerns with this PR?

FYI we are going to include this PR in the GFSv17 retrospectives, which will be starting as soon as possible.

No other concerns. I'll get to merge testing ASAP (in between meetings today, probably).

DavidNew-NOAA · 2025-11-04T16:44:38Z

Thanks @BrianCurtis-NOAA

BrianCurtis-NOAA · 2025-11-04T20:19:59Z

Generating docs for file /scratch4/NCEPDEV/nems/Brian.Curtis/git/DavidNew-NOAA/UFS_UTILS/sorc/regrid_sfc.fd/grids_IO.F90...
/scratch4/NCEPDEV/nems/Brian.Curtis/git/DavidNew-NOAA/UFS_UTILS/sorc/regrid_sfc.fd/grids_IO.F90:366: error: The following parameter of grids_io::create_grid_fv3(integer, intent(in) res_atm, character(*), intent(in) dir_fix, integer, intent(in) npets, integer, intent(in) localpet, integer, intent(in) imem_ens, type(esmf_grid), intent(out) fv3_grid) is not documented:
  parameter 'imem_ens' (warning treated as error, aborting now)
Exiting...

BrianCurtis-NOAA

Found all the rest of the places where you'll need to document the new inputs to functions.

BrianCurtis-NOAA · 2025-11-04T20:36:16Z

still missing edit to line #254.

BrianCurtis-NOAA · 2025-11-04T20:37:19Z

and sorc/regrid_sfc.fd/readin_setup.F90 line 12

CatherineThomas-NOAA · 2025-11-04T21:01:58Z

Thanks @BrianCurtis-NOAA and @DavidNew-NOAA!

DavidNew-NOAA · 2025-11-04T21:11:41Z

Thanks @BrianCurtis-NOAA ! It's been a useful education on how to do PRs for ufs_utils (reg tests, unit tests, etc)

DavidNew-NOAA added 2 commits October 14, 2025 17:41

Update

2df8472

Bugfix

f091f4e

DavidNew-NOAA mentioned this pull request Oct 29, 2025

Parallelize surface increment regridding NOAA-EMC/global-workflow#4194

Merged

22 tasks

DavidNew-NOAA marked this pull request as ready for review October 30, 2025 13:09

CatherineThomas-NOAA requested review from BrianCurtis-NOAA, CatherineThomas-NOAA and tsga October 30, 2025 15:07

Add nmem_ens to unit test namelist

d543abc

Fix regression test

bbe8a25

tsga approved these changes Nov 3, 2025

View reviewed changes

Fix unit test

908bbaa

CatherineThomas-NOAA approved these changes Nov 4, 2025

View reviewed changes

BrianCurtis-NOAA requested changes Nov 4, 2025

View reviewed changes

Comment thread sorc/regrid_sfc.fd/grids_IO.F90

Add documentation for function subroutine inputs

616c096

BrianCurtis-NOAA requested changes Nov 4, 2025

View reviewed changes

Comment thread sorc/regrid_sfc.fd/grids_IO.F90

Comment thread sorc/regrid_sfc.fd/grids_IO.F90

Comment thread sorc/regrid_sfc.fd/grids_IO.F90

Comment thread sorc/regrid_sfc.fd/readin_setup.F90

More missing documentation

7d16d0b

More missing documentation

a7d2cb9

BrianCurtis-NOAA approved these changes Nov 4, 2025

View reviewed changes

BrianCurtis-NOAA merged commit 154a367 into ufs-community:develop Nov 4, 2025
1 of 2 checks passed

BrianCurtis-NOAA added this to UFS_UTILS v1.15.0 Nov 4, 2025

BrianCurtis-NOAA moved this to Done in UFS_UTILS v1.15.0 Nov 4, 2025

Conversation

DavidNew-NOAA commented Oct 14, 2025 • edited by BrianCurtis-NOAA Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

DESCRIPTION OF CHANGES:

TESTS CONDUCTED:

DEPENDENCIES:

DOCUMENTATION:

Uh oh!

BrianCurtis-NOAA commented Oct 30, 2025

Uh oh!

BrianCurtis-NOAA commented Oct 30, 2025

Uh oh!

BrianCurtis-NOAA commented Oct 30, 2025

Uh oh!

DavidNew-NOAA commented Oct 30, 2025

Uh oh!

DavidNew-NOAA commented Oct 31, 2025

Uh oh!

BrianCurtis-NOAA commented Oct 31, 2025

Uh oh!

DavidNew-NOAA commented Oct 31, 2025

Uh oh!

BrianCurtis-NOAA commented Oct 31, 2025

Uh oh!

DavidNew-NOAA commented Oct 31, 2025

Uh oh!

DavidNew-NOAA commented Nov 3, 2025

Uh oh!

BrianCurtis-NOAA commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DavidNew-NOAA commented Nov 3, 2025

Uh oh!

BrianCurtis-NOAA commented Nov 3, 2025

Uh oh!

DavidNew-NOAA commented Nov 3, 2025

Uh oh!

BrianCurtis-NOAA commented Nov 3, 2025

Uh oh!

DavidNew-NOAA commented Nov 3, 2025

Uh oh!

BrianCurtis-NOAA commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BrianCurtis-NOAA commented Nov 3, 2025

Uh oh!

DavidNew-NOAA commented Nov 3, 2025

Uh oh!

DavidNew-NOAA commented Nov 3, 2025

Uh oh!

BrianCurtis-NOAA commented Nov 3, 2025

Uh oh!

DavidNew-NOAA commented Nov 3, 2025

Uh oh!

CatherineThomas-NOAA commented Nov 4, 2025

Uh oh!

BrianCurtis-NOAA commented Nov 4, 2025

Uh oh!

DavidNew-NOAA commented Nov 4, 2025

Uh oh!

BrianCurtis-NOAA commented Nov 4, 2025

Uh oh!

Uh oh!

BrianCurtis-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BrianCurtis-NOAA commented Nov 4, 2025

Uh oh!

BrianCurtis-NOAA commented Nov 4, 2025

Uh oh!

Uh oh!

CatherineThomas-NOAA commented Nov 4, 2025

Uh oh!

DavidNew-NOAA commented Nov 4, 2025

Uh oh!

Reviewers

DavidNew-NOAA commented Oct 14, 2025 •

edited by BrianCurtis-NOAA

Loading

BrianCurtis-NOAA commented Nov 3, 2025 •

edited

Loading

BrianCurtis-NOAA commented Nov 3, 2025 •

edited

Loading