Avoid parallel restart I/O on WCOSS2

[DavidHuber-NOAA]

Description

This disables parallel restart reads on WCOSS2 by building the UFS via build.sh instead of tests/compile.sh. This is a temporary workaround to allow C384 forecasts to run to completion.

Resolves #3589
Refs ufs-community/ufs-weather-model#2716

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)
• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this a breaking change (a change in existing functionality)? YES (Parallel restart I/O will be disabled on WCOSS2, resulting in slower runtimes for forecasts)
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? NO

How has this been tested?

Simple C384 forecast on WCOSS2.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added

[DavidHuber-NOAA]

FYI @DusanJovic-NOAA

[DavidHuber-NOAA]

Opened #3616 to reinstate parallel restart file reads once ufs-community/ufs-weather-model#2716 is addressed.

[JessicaMeixner-NOAA]

We are in the middle of scaling/timing tests for GFS & GEFS ---- so I'm pointing this PR out to @GeorgeVandenberghe-NOAA @DusanJovic-NOAA @bingfu-NOAA @NeilBarton-NOAA @CatherineThomas-NOAA @RuiyuSun - so that they make sure they have parallel restarts enabled for any timing tests. I don't think George or Dusan are using g-w builds, but others might be.

[GeorgeVandenberghe-NOAA]

George Vandenberghe is using a GW build

…

On Fri, Apr 25, 2025 at 5:09 PM Jessica Meixner ***@***.***> wrote: *JessicaMeixner-NOAA* left a comment (NOAA-EMC/global-workflow#3615) <#3615 (comment)> We are in the middle of scaling/timing tests for GFS & GEFS ---- so I'm pointing this PR out to @GeorgeVandenberghe-NOAA <https://github.com/GeorgeVandenberghe-NOAA> @DusanJovic-NOAA <https://github.com/DusanJovic-NOAA> @bingfu-NOAA <https://github.com/bingfu-NOAA> @NeilBarton-NOAA <https://github.com/NeilBarton-NOAA> @CatherineThomas-NOAA <https://github.com/CatherineThomas-NOAA> @RuiyuSun <https://github.com/RuiyuSun> - so that they make sure they have parallel restarts enabled for any timing tests. I don't think George or Dusan are using g-w builds, but others might be. — Reply to this email directly, view it on GitHub <#3615 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDS4FXONA5O5GFWBG5TNB323JT5FAVCNFSM6AAAAAB34BWE52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMZQHE3TINZYGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)

[CatherineThomas-NOAA]

Thanks for the PR @DavidHuber-NOAA. I did a quick clone/build to make sure this addressed my previous issue #3589. The model built successfully, but after there was an error and the build script failed:

 [100%] Built target ufs_model
make[1]: Leaving directory '/lfs/h2/emc/da/noscrub/catherine.thomas/git/global-workflow/dave_fix/sorc/ufs_model.fd/build_fv3_1'
/apps/spack/cmake/3.20.2/intel/19.1.3.304/utnbptm3hrf7gppztidueu4jogfgemut/bin/cmake -E cmake_progress_start /lfs/h2/emc/da/noscrub/catherine.thomas/git/global-workflow/dave_fix/sorc/ufs_model.fd/build_fv3_1/CMakeFiles 0
+ mv /lfs/h2/emc/da/noscrub/catherine.thomas/git/global-workflow/dave_fix/sorc/ufs_model.fd/build_fv3_1/ufs_model tests/fv3_1.exe
+ [[ NO == \Y\E\S ]]
+ mv ./tests/fv3_1.exe ./tests/gfs_model.x
+ [[ ! -f ./tests/modules.ufs_model.lua ]]
+ mv ./tests/modules.fv3_1.lua ./tests/modules.ufs_model.lua
mv: cannot stat './tests/modules.fv3_1.lua': No such file or directory

[CatherineThomas-NOAA]

Also, I can confirm that even though the build script failed, the executable was created beforehand and running with that executable allowed my previously failed test (C384_atm3DVar) to succeed.

[DavidHuber-NOAA]

@CatherineThomas-NOAA Thanks for the test and letting me know about my slop. I put in a fix just now and am testing the build presently.

@GeorgeVandenberghe-NOAA @JessicaMeixner-NOAA I will add in an option to compile using compile.sh (i.e. enabling parallel restart reads).

[DavidHuber-NOAA]

My fix did not work... Continuing to diagnose.

[DavidHuber-NOAA]

@GeorgeVandenberghe-NOAA @JessicaMeixner-NOAA I've added an option to build the UFS with parallel restart I/O enabled. This option will only work with the build_all.sh and build_ufs.sh scripts (not build_compute.sh, which has fixed build parameters). To use it, you will just need to pass -p to the script.

@CatherineThomas-NOAA I have fixed the build issues now and my test build was successful.

I will go ahead and start a full suite of CI tests, including the weekly tests, on WCOSS2.

[DavidHuber-NOAA]

All tests passed except the weekly C384_S2SWA test due to missing initial conditions. This test has not been run in quite some time, so this is not a big surprise.

@KateFriedman-NOAA is it possible to have the C384_S2SWA ICs uploaded to WCOSS2?

@CatherineThomas-NOAA should this test be run to consider this PR a success?

[KateFriedman-NOAA]

is it possible to have the C384_S2SWA ICs uploaded to WCOSS2?

@DavidHuber-NOAA I only have the ice IC file for this case. I don't have the wave ICs (link in older timestamp is to empty folder) and I don't have any atmos ICs for this case. If someone from CPC or elsewhere can provide the ICs I can add them to the staged sets.

[DavidHuber-NOAA]

@KateFriedman-NOAA Thanks for checking! I opened up #3618 to track the addition of those ICs. Feel free to ping appropriate folks in that issue.

Setting the WCOSS2-CI-Passed label as all other tests passed.

I believe this PR is ready for final reviews.

[CatherineThomas-NOAA]

@CatherineThomas-NOAA should this test be run to consider this PR a success?

@DavidHuber-NOAA - The test you mentioned is for the GEFS. Since that test hasn't been run in some time and this isn't a new error, I think it's fine to move on with this PR from my perspective so long as the original error is fixed and no regularly run tests are failing. Thanks for opening the issue to get that test back up and running!

[CatherineThomas-NOAA]

Confirming that I no longer get any errors from the build script with your latest fix @DavidHuber-NOAA. Thank you for adding the -p option so that our scaling work can continue and for opening the companion issue to reenable this option once the ufs issue is addressed.

[aerorahul]

lgtm