Use MPI_Bcast instead of multiple p2p messages to update nest from parent

[dkokron]

PR Author Checklist:

• I have linked PR's from all sub-components involved in section below.

• I am confirming reviews are completed in ALL sub-component PR's.

• I have run the full RT suite on either Hera/Hercules AND have attached the log to this PR below this line:

  • LOG:
    RegressionTests_hercules.log

• I have added the list of all failed regression tests to "Anticipated changes" section.

• I have filled out all sections of the template.

Performance profiling of a HAFS case on NOAA systems revealed significant of time was spent in fill_nested_grid_cpl(). The fill_nested_grid_cpl() routine from FV3/atmos_cubed_sphere/driver/fvGFS/atmosphere.F90 is showing up as a performance bottleneck. This routine gathers a global SST field (6,336,000 bytes) onto rank 0, then broadcasts that field to all ranks in the nest. The code uses point-to-point (p2p) messages (Isend/Recv) from rank 0 to the nest ranks. This communication pattern is maxing out the SlingShot-10 link on the first node resulting in a .15s hit every fifth time step.

The proposed fix is to modify the relevant FV3 code to use a single MPI_Bcast (via mpp_broadcast()) instead of multiple point-to-point messages (via mpp_send/mpp_recv). The use of mpp_broadcast depends on a fix to FMS that was merged on 16 June and is available in version 2023.02 of that package.

• Replace current_peset_num with 'n' which should be the active pelist. NOAA-GFDL/FMS#1246

This PR depends on

• Use MPI_Bcast instead of multiple p2p messages to update nest from parent NOAA-GFDL/GFDL_atmos_cubed_sphere#272

This PR also depends on

• Use MPI_Bcast instead of multiple p2p messages to update nest from parent NOAA-EMC/ufsatm#743

I ran the UFS regression suite on acorn and cactus. Both runs resulted in "REGRESSION TEST WAS SUCCESSFUL". I do not have access to Hera/Hercules.

This change is zero-diff. No need to update baselines.

Commit Message

• use single MPI_Bcast (via mpp_broadcast()) instead of multiple point-to-point messages (via mpp_send/mpp_recv)
• address issue maxing out the SlingShot-10 link on the first node resulting in a .15s hit every fifth time step

Subcomponents involved:

• AQM
• CDEPS
• CICE
• CMEPS
• CMakeModules
• FV3
• GOCART
• HYCOM
• MOM6
• NOAHMP
• WW3
• stochastic_physics
• none

Anticipated Changes

Input data

• No changes are expected to input data.
• Changes are expected to input data:
  • New input data.
  • Updated input data.

Regression Tests:

• No changes are expected to any regression test.
• Changes are expected to the following tests:

Tests effected by changes in this PR:

Libraries

• Not Needed
• Needed
  • Create separate issue in JCSDA/spack-stack asking for update to library. Include library name, library version.
  • Add issue link from JCSDA/spack-stack following this item

Code Managers Log

• This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR.
• Move new/updated input data on RDHPCS Hera and propagate input data changes to all supported systems.
  • N/A

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • Jet
  • Gaea
  • Derecho
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
  • Completed
• opnReqTest
  • N/A
  • Log attached to comment

[jkbk2004]

@dkokron can you sync up your branches? We plan to work on this pr by combining with #2066 tomorrow.

[BrianCurtis-NOAA]

@dkokron you'll want to make sure to pull from ufs-community/ufs-weather-model develop branch and then update your submodule hashes. This includes in your FV3 PR as well bring the updates into UFSWM.

[jkbk2004]

@dkokron I think branches got synced ok. Can you check files changed?

[BrianCurtis-NOAA]

Skipping WCOSS2 as it's down today. Will be back at 20Z if PR gets delayed past that.

[zach1221]

@dkokron @jkbk2004 fv3atm is merged. Please update hash and revert gitmodule url.
Hash: NOAA-EMC/ufsatm@6c2b775