Fix hang bug by updating MPI bcast to be from 0 not IAPROC-1 + Restart configurable path debugging for global-workflow

[JessicaMeixner-NOAA]

Pull Request Summary

When trying to use the save points, a hang on wcoss2 was discovered, this was due to a bug.

• Resolved an issue preventing the successful reading of WW3 restart files when using a configurable restart path in the ufs-weather-model and global-workflow. From PR (Restart configurable path debugging for global-workflow #1420)

Description

Updating the broadcast calls to be from 0 instead of IAPROC-1 in w3iopo for when we have saved weight files for the unstructured grid case.

A PR to develop was made - #1430

Closes #1420 as it was added here

Issue(s) addressed

Refs #1350
Refs ufs-community/ufs-weather-model#2709

Commit Message

Fix hang bug by updating MPI bcast to be from 0 not IAPROC-1

Check list

• Branch is up to date with the authoritative repository (NOAA-EMC) develop branch.
• Checked the checklist for a developer submitting to develop.
• If a version number update is required, checked the updating version number checklist.
• If a new feature was added, a regression test for testing the new feature is added.

Testing

• How were these changes tested? These were tested in standalone ww3 regtests and in ufs-weather-model RT tests
• Are the changes covered by regression tests? (If not, why? Do new tests need to be added?) yes - although the hang was seen on larger tests mostly.
• Have the matrix regression tests been run (if yes, please note HPC and compiler)? hera intel
• Please indicate the expected changes in the regression test output, (Note the list of known non-identical tests.)
  No expected answer changes.
• Please provide the summary output of matrix.comp (matrix.Diff.txt, matrixCompFull.txt and matrixCompSummary.txt):

**********************************************************************
********************* non-identical cases ****************************
**********************************************************************
mww3_test_03/./work_PR3_UQ_MPI_e_c                     (1 files differ)
mww3_test_03/./work_PR3_UNO_MPI_e                     (1 files differ)
mww3_test_03/./work_PR2_UQ_MPI_e                     (1 files differ)
mww3_test_03/./work_PR2_UNO_MPI_e                     (1 files differ)
mww3_test_03/./work_PR2_UNO_MPI_d2                     (17 files differ)
mww3_test_03/./work_PR1_MPI_d2                     (13 files differ)
mww3_test_03/./work_PR3_UNO_MPI_d2_c                     (15 files differ)
mww3_test_03/./work_PR3_UQ_MPI_d2_c                     (17 files differ)
mww3_test_03/./work_PR3_UNO_MPI_d2                     (12 files differ)
mww3_test_03/./work_PR2_UQ_MPI_d2                     (15 files differ)
mww3_test_03/./work_PR3_UQ_MPI_e                     (1 files differ)
mww3_test_03/./work_PR3_UNO_MPI_e_c                     (1 files differ)
mww3_test_03/./work_PR3_UQ_MPI_d2                     (16 files differ)
mww3_test_09/./work_MPI_ASCII                     (0 files differ)
ww3_tp2.10/./work_MPI_OMPH                     (7 files differ)
ww3_tp2.14/./work_OASACM5                     (1 files differ)
ww3_tp2.16/./work_MPI_OMPH                     (4 files differ)
ww3_tp2.6/./work_ST4_ASCII                     (0 files differ)
ww3_ufs1.3/./work_a                     (3 files differ)

ww3_tp2.14/./work_OASACM5 diffs in OUTPUT_TOY.txt which is unexpected but isn't meaningful from what I can tell:

7c7,8
<  ===========================================================================
---
>  APPLE partitioning
> ========================================================

matrixCompFull.txt
matrixCompSummary.txt
matrixDiff.txt

UFS PR: ufs-community/ufs-weather-model#2737

[sbanihash]

I have reviewed this PR and approve the changes. Matrix comparison files are attached.

My ww3_tp2.14/./work_OASACM5 test shows no file differences. So I am guessing there was just some strange thing happening in Jessica's test.

matrixDiff.txt
matrixCompSummary.txt
matrixCompFull.txt

[JessicaMeixner-NOAA]

matrixCompFull.txt
matrixCompSummary.txt
matrixDiff.txt