Fix GFSv17 Restart IAU Reproducibility by Adjusting WW3 Internal Timestep by mingchen-NOAA · Pull Request #2784 · ufs-community/ufs-weather-model

mingchen-NOAA · 2025-06-23T16:18:59Z

Commit Queue Requirements:

Fill out all sections of this template.
All sub component pull requests have been reviewed by their code managers.
Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
Commit 'test_changes.list' from previous step

Description:

This pull request resolves the GFSv17 restart IAU reproducibility issue by modifying the WW3 internal timestep in ww3_grid.inp.exp.global_270k from 720. 360. 360. 30. to 900. 450. 450. 30. to ensure the ITIME parameter is consistent between control and restart runs. The WW3 input files were regenerated accordingly. A new test configuration file tests/tests/cpld_restart_gfsv17_iau was added, and tests/rt.conf was updated to include this restart IAU test case while removing the GFSv17 no wave test cases. Additionally, tests/fv3_conf/cpld_control_run.IN was modified to address an issue with creating the pnt_wght.ww3.nc file for the restart IAU test, and tests/rt.sh was updated to point to the newly created input directory.

The new input directory on Hera:
/scratch4/NCEPDEV/marine/Ming.Chen/ufs_iau_timefix/WW3_input_data_20250715

A full rt tests were conducted and following changes were found due to change the input timesteps.

cpld_control_gfsv17 intel
cpld_control_gfsv17_iau intel
cpld_restart_gfsv17 intel
cpld_restart_gfsv17_iau intel
cpld_mpi_gfsv17 intel
cpld_control_sfs intel
cpld_debug_gfsv17 intel

Commit Message:

* UFSWM - Updated WW3 timestep input and created cpld_restart_gfsv17_iau test case

Priority:

Normal

Git Tracking

UFSWM:

Add GFSv17 IAU restart tests #2306

Sub component Pull Requests:

None

UFSWM Blocking Dependencies:

None

Documentation:

This PR requires a documentation update, and the WM User's Guide has been updated based on the changes in this PR.
This PR requires a documentation update, and a WM issue has been opened to track the need for a documentation update; a person responsible for submitting the update has been assigned to the issue (link issue).
No documentation update is required for this PR (please explain).

Changes

Regression Test Changes (Please commit test_changes.list):

PR Adds New Tests/Baselines.
PR Updates/Changes Baselines.

Input data Changes:

/scratch4/NCEPDEV/marine/Ming.Chen/ufs_iau_timefix/WW3_input_data_20250715

Library Changes/Upgrades:

No Updates

Testing Log:

DeniseWorthen · 2025-07-14T19:50:20Z

@mingchen-NOAA Please add a Hera location where the new WW3 input-data can be found, it will be sync'd from that location to all RDHPCS.

Also, if older WW3-input-data can be removed, it might be a good time to clean them up (?). Currently we're retaining input from 2022 and 2024.

WW3_input_data_20220624
WW3_input_data_20240214
WW3_input_data_20250114
WW3_input_data_20250212
WW3_input_data_20250225

mingchen-NOAA · 2025-07-14T20:29:33Z

The input data for this fix on Hera is defined in the modified rt.sh script (changes in rt.sh) or
/scratch2/NCEPDEV/marine/Ming.Chen/ufs-weather-model/ufs_IAU/ufs_iau_timefix/ww3_input_timefix

I will check input data and remove the unused old files. Thank you!

DeniseWorthen · 2025-07-14T20:39:54Z

 RUN | cpld_mpi_gfsv17                                   | - noaacloud                          |          |
-RUN | cpld_control_gfsv17_nowav_iau                     | - noaacloud                          | baseline | cpld_control_gfsv17
-RUN | cpld_restart_gfsv17_nowav_iau                     | - noaacloud                          |          | cpld_control_gfsv17_nowav_iau
+#RUN | cpld_control_gfsv17_nowav_iau                     | - noaacloud                          | baseline | cpld_control_gfsv17


I believe the expectation is that the nowav_iau tests would be removed.

Yes. We will remove the no-wave tests b/c we have created a cpld_restart_gfsv17_iau test.

But shouldn't both the actual tests be removed and the rt.conf fixed?

I will fix the rt.conf. Thanks!

Now I remove nowav_iau tests from rt.conf. Thank you!

DeniseWorthen · 2025-07-14T20:41:06Z

@mingchen-NOAA Could you add the new input-data path (to make it explicit for the EPIC CMs) to the PR information? Thanks!

mingchen-NOAA · 2025-07-14T20:41:53Z

Yes. I will fix it with the input path.

DeniseWorthen · 2025-08-06T14:01:46Z

@mingchen-NOAA The same itime bug is the reason that the current ultralow configurations do not restart reproduce. See #2652. I would like to include the fix in this PR if you would agree. It would mean

updating mod_def.uglo_900km. I've tested timesteps 1800. 900. 900. 30. and 1800. 1800. 1800. 30. and both give restart repro.
uncommenting the LIST_FILES in the ultralow restart tests.
retesting; all the ultralow configs will then require new baselines in addition to the IAU tests.

If you agree, I can prepare a PR which you can merge w/ your existing UWM PR.

mingchen-NOAA · 2025-08-06T14:22:34Z

I’m currently preparing a code fix PR to address this issue as a long-term solution, so that input changes won’t be necessary. I plan to submit the new PR today. If you think it would be more efficient, we can combine this with the updates you’ve outlined so they are included in a single PR.

DeniseWorthen · 2025-08-06T14:26:01Z

@mingchen-NOAA Thanks. So do you intend to use this PR (with the timestep change) or close this and create a new one?

mingchen-NOAA · 2025-08-06T14:31:45Z

I prefer to create a new one. Could you prepare a PR so I can merge it and test it together with my code fix? We can start from there. Sounds good?

DeniseWorthen · 2025-08-06T14:34:16Z

OK, sounds good. I knew you'd need a WW3 update, so I figured you would be creating a new one. I'll create one and tag it here. Thanks.

mingchen-NOAA · 2025-08-06T14:37:02Z

Or, you can point me to your branch with the test configurations so I can run the tests directly.

DeniseWorthen · 2025-08-06T14:39:46Z

Here is my branch enabling the ultralow restarts

https://github.com/DeniseWorthen/ufs-weather-model/tree/feature/enable_ultralow_restarts

mingchen-NOAA · 2025-08-06T14:40:25Z

Thank you! I will test it with my code fix.

DeniseWorthen · 2025-08-06T18:22:17Z

Oh shoot. No, that won't work because we can't alter an existing input-data directory. We can add new input, but we cannot change existing input. (For WW3, we just create a new sub-directory but that is the only case where we carry a separate input-data.)

I was trying to avoid new input-data in this go-around, since I want to do the new input-data directory once we have the V2 bug in the ATM restarts fixed; that will come in with my SFS C192mx025 test update and who knows how many weeks it will be before that gets to the top of the Q.

Could we just fix the ultralow tests for now by modifying the mod_def? These tests run so fast, I don't think it matters much if we make WW3 take a little longer.

mingchen-NOAA · 2025-08-06T18:26:33Z

No worries. I fixed my code to separate the write and read control parameter so for the warmstart cases, I can make it write only.

dpsarmie · 2025-08-18T16:37:13Z

@mingchen-NOAA Is this PR still needed or will #2851 replace this PR?
#2851 is scheduled to be next in the commit queue and I wanted to double-check on this PR.

mingchen-NOAA · 2025-08-18T16:44:50Z

#2851 replaces this one. Thank you for checking!

dpsarmie · 2025-08-18T16:56:14Z

Ok then I will close this PR for now. If you need to reopen it, feel free to do so.

mingchen-NOAA · 2025-08-18T16:56:52Z

Thank you very much!

mingchen-NOAA added 5 commits June 17, 2025 07:35

create cpld_restart_gfsv17_iau test and remove no-wave cases

55fba83

Merge branch 'ufs-community:develop' into ufs_gfsv17_restart_iau

71bcae5

Merge branch 'ufs-community:develop' into ufs_gfsv17_restart_iau

0dab951

update rt.sh for modified time steps and rt tests results

36ba37e

merge to to develop

973907b

junwang-noaa mentioned this pull request Jul 3, 2025

[DRAFT] GFSv17 iau restart test #2404

Closed

14 tasks

DeniseWorthen added Baseline Updates Current baselines will be updated. Changes Existing Input Data Existing input data will be changed. A new input-data-YYYYMMDD directory must be created. labels Jul 14, 2025

DeniseWorthen reviewed Jul 14, 2025

View reviewed changes

mingchen-NOAA added 2 commits July 14, 2025 21:17

update rt.conf to remove the nowav_iau tests

a09f419

move inputdata to scratch4 and remove nowave tests

4f89a3d

dpsarmie closed this Aug 18, 2025

mingchen-NOAA deleted the ufs_gfsv17_restart_iau branch November 21, 2025 15:30

Conversation

mingchen-NOAA commented Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Commit Queue Requirements:

Description:

Commit Message:

Priority:

Git Tracking

UFSWM:

Sub component Pull Requests:

UFSWM Blocking Dependencies:

Documentation:

Changes

Regression Test Changes (Please commit test_changes.list):

Input data Changes:

Library Changes/Upgrades:

Testing Log:

Uh oh!

DeniseWorthen commented Jul 14, 2025

Uh oh!

mingchen-NOAA commented Jul 14, 2025

Uh oh!

DeniseWorthen Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

mingchen-NOAA Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

DeniseWorthen Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mingchen-NOAA Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

mingchen-NOAA Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

DeniseWorthen commented Jul 14, 2025

Uh oh!

mingchen-NOAA commented Jul 14, 2025

Uh oh!

DeniseWorthen commented Aug 6, 2025

Uh oh!

mingchen-NOAA commented Aug 6, 2025

Uh oh!

DeniseWorthen commented Aug 6, 2025

Uh oh!

mingchen-NOAA commented Aug 6, 2025

Uh oh!

DeniseWorthen commented Aug 6, 2025

Uh oh!

mingchen-NOAA commented Aug 6, 2025

Uh oh!

DeniseWorthen commented Aug 6, 2025

Uh oh!

mingchen-NOAA commented Aug 6, 2025

Uh oh!

DeniseWorthen commented Aug 6, 2025

Uh oh!

mingchen-NOAA commented Aug 6, 2025

Uh oh!

dpsarmie commented Aug 18, 2025

Uh oh!

mingchen-NOAA commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dpsarmie commented Aug 18, 2025

Uh oh!

mingchen-NOAA commented Aug 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mingchen-NOAA commented Jun 23, 2025 •

edited

Loading

DeniseWorthen Jul 14, 2025 •

edited

Loading

mingchen-NOAA commented Aug 18, 2025 •

edited

Loading