Fix Restart IAU and Ultra-Low Reproducibility Issues by Adding Control Parameter to Restart File

[mingchen-NOAA]

Pull Request Summary

This PR addresses issues with restart IAU and ultra-low reproducibility in UFS-WM by introducing a new control parameter in the restart file. This parameter ensures consistent behavior between 'control' and 'restart' runs when IAU or ultra-low features are enabled, improving reproducibility and model consistency.

Description

The restart reproducibility issue in UFS-WM when using IAU or Ultra-low feature has been traced to the following line of code:
https://github.com/NOAA-EMC/WW3/blob/develop/model/src/w3pro3md.F90#L1851
In the control run (IAU or warmstart), the time step counter, ITIME is even at the restart time. However, in the restart run, ITIME is reinitialized and starts from 1, making it odd at the same point in simulation. This mismatch causes minor but consistent differences between the 'control' and 'restart' runs.

This PR introduces a new feature that enables writing, reading, or both writing and reading a control parameter in the restart file. The feature addresses issues related to restart IAU and ultra-low reproducibility in UFS-WM.

In detail, this PR modifies wav_restart_mod.F90 to enable an optional feature for writing, reading, or both writing and reading a control parameter in the restart file. This enhancement ensures consistent behavior between control and restart runs in the UFS-WM, particularly when IAU or ultra-low reproducibility features are activated.

A new control parameter, 'ITSTEP', is declared and initialized in the W3ADATMD module. It replaces the original 'ITIME' variable to control run-time execution order in the subroutines W3KTP2 and W3KTP3, located in w3pro2md.F90 and w3pro3md.F90, respectively.

Issue(s) addressed

Addressing issue #2306 and issue #2652

Commit Message

Add optional control parameter to restart file to ensure consistent IAU and ultra-low reproducibility behavior

Check list

• Branch is up to date with the authoritative repository (NOAA-EMC) develop branch.
• Checked the checklist for a developer submitting to develop.
• If a version number update is required, checked the updating version number checklist.
• If a new feature was added, a regression test for testing the new feature is added.

Testing

• How were these changes tested? Tested using a full RT tests in UFS-WM
• Are the changes covered by regression tests? (If not, why? Do new tests need to be added?)
• Have the matrix regression tests been run (if yes, please note HPC and compiler)? The UFS-WM RT tests ran on Hercules Intel
• Please indicate the expected changes in the regression test output, (Note the list of known non-identical tests.)
• Please provide the summary output of matrix.comp (matrix.Diff.txt, matrixCompFull.txt and matrixCompSummary.txt):

[NickSzapiro-NOAA]

So in model/src/wav_comp_nuopc.F90 there is a TimeOffset = currTime - startTime that could be used to match steps without any extra settings. Is the problem that there is no similar "startTime" for standalone WW3 for restart run?

[mingchen-NOAA]

It does not record a start time or similar and then offset from start time. In my understanding, using startTime based TimeOffset may not be sufficient/straightforward to match time step counter between control and restart. We may need a global reference time to define control parameter (time step counter or similar to ITIME).

Writing a control parameter in restart file as an optional feature provides a more straightforward solution. We can use it as needed.

[jkbk2004]

all test are done ok at ufs-community/ufs-weather-model#2851. @mingchen-NOAA @JessicaMeixner-NOAA can you merge this pr?