Skip to content

Fix Restart IAU and Ultra-Low Reproducibility Issues by Adding Control Parameter to Restart File #2851

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jkbk2004 merged 19 commits into from
Aug 22, 2025
Merged

Fix Restart IAU and Ultra-Low Reproducibility Issues by Adding Control Parameter to Restart File #2851

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
a18e55e
enable restart tests for ultralow configurations
DeniseWorthen Aug 6, 2025
f047154
fix GFSv17 restart IAU reproducibility issue by fixing the code
Aug 6, 2025
785e38d
fix reproducibility issue for ultralow configuration
Aug 6, 2025
e045d98
attached log and test changes list
Aug 7, 2025
218d5b2
fix the ww3 branch
mingchen-NOAA Aug 19, 2025
49e04c3
Update bl_date.conf
jkbk2004 Aug 19, 2025
bc5f1a4
Update WW3 submodule to ufs_wm/iau_codefix @ 0929f9
mingchen-NOAA Aug 20, 2025
6b23be1
ORT Jobs Completed.
epic-cicd-jenkins Aug 20, 2025
98ad623
ORT Jobs Completed.
epic-cicd-jenkins Aug 21, 2025
93d300c
add hercules RT log: passed
jkbk2004 Aug 21, 2025
6479fff
add gaeac6 RT log passed
FernandoAndrade-NOAA Aug 21, 2025
e610280
ORT Jobs Completed.
epic-cicd-jenkins Aug 21, 2025
86cf71d
add derecho RT log: passed
jkbk2004 Aug 21, 2025
f6643cb
add orion RT log: WLCLK adjustment
jkbk2004 Aug 21, 2025
738e799
add hera RT log passed
FernandoAndrade-NOAA Aug 21, 2025
7ea65d8
WCOSS2 RT: Passed
BrianCurtis-NOAA Aug 21, 2025
e387353
Acorn RT: Passed
BrianCurtis-NOAA Aug 21, 2025
5fcc69d
add ursa RT log passed
FernandoAndrade-NOAA Aug 21, 2025
8c7d763
update WW3 to hash 2a96637
mingchen-NOAA Aug 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion WW3
Open in desktop
Submodule WW3 updated 5 files
+4 −2 model/src/w3adatmd.F90
+2 −2 model/src/w3pro2md.F90
+2 −2 model/src/w3pro3md.F90
+5 −1 model/src/w3wavemd.F90
+31 −7 model/src/wav_restart_mod.F90
2 changes: 1 addition & 1 deletion tests/bl_date.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
export BL_DATE=20250815
export BL_DATE=20250819
6 changes: 5 additions & 1 deletion tests/fv3_conf/cpld_control_run.IN
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,11 @@ else
RFILE=ufs.cpld.ww3.r.${RESTART_FILE_SUFFIX_SECS}.nc
cp ../${DEP_RUN}${SUFFIX}/${RFILE} .
if [[ $WW3_DOMAIN == 'global_270k' ]]; then
cp ../${DEP_RUN}${SUFFIX}/out.pnt_wght.ww3.nc pnt_wght.ww3.nc
if [[ -f ../${DEP_RUN}${SUFFIX}/out.pnt_wght.ww3.nc ]]; then
cp ../${DEP_RUN}${SUFFIX}/out.pnt_wght.ww3.nc pnt_wght.ww3.nc
else
cp ../${DEP_RUN}${SUFFIX}/pnt_wght.ww3.nc pnt_wght.ww3.nc
fi
fi
fi

Expand Down
48 changes: 24 additions & 24 deletions tests/logs/OpnReqTests_control_p8_hera.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Mon Aug 18 18:07:13 UTC 2025
Thu Aug 21 03:41:30 UTC 2025
Start Operation Requirement Test


baseline dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-stmp/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_bit_base_gnu
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_674980/bit_base_bit_base
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_89154/bit_base_bit_base
Checking test bit_base results ....
Moving baseline bit_base files ....
Moving sfcf000.nc .........OK
Expand Down Expand Up @@ -51,14 +51,14 @@ Moving baseline bit_base files ....
Moving RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Moving RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 249.802448
0: The maximum resident set size (KB) = 1556932
0: The total amount of wall time = 253.174378
0: The maximum resident set size (KB) = 1553140

Test bit_base PASS


baseline dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-stmp/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_dbg_base_gnu
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_674980/dbg_base_dbg_base
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_89154/dbg_base_dbg_base
Checking test dbg_base results ....
Moving baseline dbg_base files ....
Moving sfcf000.nc .........OK
Expand Down Expand Up @@ -106,14 +106,14 @@ Moving baseline dbg_base files ....
Moving RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Moving RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 886.391999
0: The maximum resident set size (KB) = 1517516
0: The total amount of wall time = 868.268901
0: The maximum resident set size (KB) = 1539204

Test dbg_base PASS


baseline dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-stmp/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_674980/dcp_dcp
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_89154/dcp_dcp
Checking test dcp results ....
Comparing sfcf000.nc .....USING NCCMP......OK
Comparing sfcf021.nc .....USING NCCMP......OK
Expand Down Expand Up @@ -160,14 +160,14 @@ Checking test dcp results ....
Comparing RESTART/20210323.060000.sfc_data.tile5.nc .....USING NCCMP......OK
Comparing RESTART/20210323.060000.sfc_data.tile6.nc .....USING NCCMP......OK

0: The total amount of wall time = 231.125291
0: The maximum resident set size (KB) = 1519236
0: The total amount of wall time = 232.539896
0: The maximum resident set size (KB) = 1549252

Test dcp PASS


baseline dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-stmp/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_674980/mpi_mpi
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_89154/mpi_mpi
Checking test mpi results ....
Comparing sfcf000.nc .....USING NCCMP......OK
Comparing sfcf021.nc .....USING NCCMP......OK
Expand Down Expand Up @@ -214,14 +214,14 @@ Checking test mpi results ....
Comparing RESTART/20210323.060000.sfc_data.tile5.nc .....USING NCCMP......OK
Comparing RESTART/20210323.060000.sfc_data.tile6.nc .....USING NCCMP......OK

0: The total amount of wall time = 226.605230
0: The maximum resident set size (KB) = 1521372
0: The total amount of wall time = 225.718905
0: The maximum resident set size (KB) = 1532676

Test mpi PASS


baseline dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-stmp/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_674980/rst_rst
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_89154/rst_rst
Checking test rst results ....
Comparing sfcf000.nc .....USING NCCMP......OK
Comparing sfcf021.nc .....USING NCCMP......OK
Expand Down Expand Up @@ -268,14 +268,14 @@ Checking test rst results ....
Comparing RESTART/20210323.060000.sfc_data.tile5.nc .....USING NCCMP......OK
Comparing RESTART/20210323.060000.sfc_data.tile6.nc .....USING NCCMP......OK

0: The total amount of wall time = 227.082606
0: The maximum resident set size (KB) = 1534680
0: The total amount of wall time = 231.150086
0: The maximum resident set size (KB) = 1522416

Test rst PASS


baseline dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-stmp/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_674980/std_base_std_base
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_89154/std_base_std_base
Checking test std_base results ....
Moving baseline std_base files ....
Moving sfcf000.nc .........OK
Expand Down Expand Up @@ -323,14 +323,14 @@ Moving baseline std_base files ....
Moving RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Moving RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 228.475472
0: The maximum resident set size (KB) = 1534432
0: The total amount of wall time = 228.285413
0: The maximum resident set size (KB) = 1534768

Test std_base PASS


baseline dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-stmp/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_674980/thr_thr
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_89154/thr_thr
Checking test thr results ....
Comparing sfcf000.nc .....USING NCCMP......OK
Comparing sfcf021.nc .....USING NCCMP......OK
Expand Down Expand Up @@ -377,11 +377,11 @@ Checking test thr results ....
Comparing RESTART/20210323.060000.sfc_data.tile5.nc .....USING NCCMP......OK
Comparing RESTART/20210323.060000.sfc_data.tile6.nc .....USING NCCMP......OK

0: The total amount of wall time = 229.106188
0: The maximum resident set size (KB) = 1533444
0: The total amount of wall time = 229.666130
0: The maximum resident set size (KB) = 1532708

Test thr PASS

OPERATION REQUIREMENT TEST WAS SUCCESSFUL
Mon Aug 18 19:12:11 UTC 2025
Elapsed time: 01h:04m:59s. Have a nice day!
Thu Aug 21 04:44:43 UTC 2025
Elapsed time: 01h:03m:13s. Have a nice day!
24 changes: 12 additions & 12 deletions tests/logs/OpnReqTests_cpld_control_nowave_noaero_p8_hera.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Mon Aug 18 16:43:56 UTC 2025
Thu Aug 21 02:17:44 UTC 2025
Start Operation Requirement Test


baseline dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-stmp/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/cpld_control_c96_noaero_p8_dbg_base_gnu
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_610305/dbg_base_dbg_base
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_3706910/dbg_base_dbg_base
Checking test dbg_base results ....
Moving baseline dbg_base files ....
Moving sfcf021.tile1.nc .........OK
Expand Down Expand Up @@ -66,14 +66,14 @@ Moving baseline dbg_base files ....
Moving RESTART/iced.2021-03-23-21600.nc .........OK
Moving RESTART/ufs.cpld.cpl.r.2021-03-23-21600.nc .........OK

0: The total amount of wall time = 2266.771843
0: The maximum resident set size (KB) = 1658288
0: The total amount of wall time = 2306.884403
0: The maximum resident set size (KB) = 1635924

Test dbg_base PASS


baseline dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-stmp/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/cpld_control_c96_noaero_p8_std_base_gnu
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_610305/rst_rst
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_3706910/rst_rst
Checking test rst results ....
Comparing sfcf021.tile1.nc .....USING NCCMP......OK
Comparing sfcf021.tile2.nc .....USING NCCMP......OK
Expand Down Expand Up @@ -135,14 +135,14 @@ Checking test rst results ....
Comparing RESTART/iced.2021-03-23-21600.nc .....USING NCCMP......OK
Comparing RESTART/ufs.cpld.cpl.r.2021-03-23-21600.nc .....USING NCCMP......OK

0: The total amount of wall time = 586.331429
0: The maximum resident set size (KB) = 1651468
0: The total amount of wall time = 578.175631
0: The maximum resident set size (KB) = 1644736

Test rst PASS


baseline dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-stmp/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/cpld_control_c96_noaero_p8_std_base_gnu
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_610305/std_base_std_base
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_3706910/std_base_std_base
Checking test std_base results ....
Moving baseline std_base files ....
Moving sfcf021.tile1.nc .........OK
Expand Down Expand Up @@ -205,11 +205,11 @@ Moving baseline std_base files ....
Moving RESTART/iced.2021-03-23-21600.nc .........OK
Moving RESTART/ufs.cpld.cpl.r.2021-03-23-21600.nc .........OK

0: The total amount of wall time = 581.153482
0: The maximum resident set size (KB) = 1638712
0: The total amount of wall time = 574.263109
0: The maximum resident set size (KB) = 1633320

Test std_base PASS

OPERATION REQUIREMENT TEST WAS SUCCESSFUL
Mon Aug 18 18:07:12 UTC 2025
Elapsed time: 01h:23m:17s. Have a nice day!
Thu Aug 21 03:41:30 UTC 2025
Elapsed time: 01h:23m:47s. Have a nice day!
30 changes: 15 additions & 15 deletions tests/logs/OpnReqTests_regional_control_hera.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Mon Aug 18 15:50:25 UTC 2025
Thu Aug 21 01:40:16 UTC 2025
Start Operation Requirement Test


baseline dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-stmp/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/regional_control_bit_base_gnu
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_571773/bit_base_bit_base
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_3658851/bit_base_bit_base
Checking test bit_base results ....
Moving baseline bit_base files ....
Moving dynf000.nc .........OK
Expand All @@ -15,14 +15,14 @@ Moving baseline bit_base files ....
Moving NATLEV.GrbF00 .........OK
Moving NATLEV.GrbF06 .........OK

0: The total amount of wall time = 613.010482
0: The maximum resident set size (KB) = 900360
0: The total amount of wall time = 609.719983
0: The maximum resident set size (KB) = 891208

Test bit_base PASS


baseline dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-stmp/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/regional_control_std_base_gnu
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_571773/dcp_dcp
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_3658851/dcp_dcp
Checking test dcp results ....
Comparing dynf000.nc .....USING NCCMP......OK
Comparing dynf006.nc .....USING NCCMP......OK
Expand All @@ -33,14 +33,14 @@ Checking test dcp results ....
Comparing NATLEV.GrbF00 .....USING CMP......OK
Comparing NATLEV.GrbF06 .....USING CMP......OK

0: The total amount of wall time = 489.331543
0: The maximum resident set size (KB) = 856468
0: The total amount of wall time = 490.556289
0: The maximum resident set size (KB) = 852748

Test dcp PASS


baseline dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-stmp/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/regional_control_std_base_gnu
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_571773/std_base_std_base
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_3658851/std_base_std_base
Checking test std_base results ....
Moving baseline std_base files ....
Moving dynf000.nc .........OK
Expand All @@ -52,14 +52,14 @@ Moving baseline std_base files ....
Moving NATLEV.GrbF00 .........OK
Moving NATLEV.GrbF06 .........OK

0: The total amount of wall time = 490.420374
0: The maximum resident set size (KB) = 850216
0: The total amount of wall time = 492.090808
0: The maximum resident set size (KB) = 859292

Test std_base PASS


baseline dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-stmp/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/regional_control_std_base_gnu
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_571773/thr_thr
working dir = /scratch3/NCEPDEV/stmp/role.epic/ORT-ptmp/role.epic/FV3_OPNREQ_TEST/opnReqTest_3658851/thr_thr
Checking test thr results ....
Comparing dynf000.nc .....USING NCCMP......OK
Comparing dynf006.nc .....USING NCCMP......OK
Expand All @@ -70,11 +70,11 @@ Checking test thr results ....
Comparing NATLEV.GrbF00 .....USING CMP......OK
Comparing NATLEV.GrbF06 .....USING CMP......OK

0: The total amount of wall time = 486.349947
0: The maximum resident set size (KB) = 865572
0: The total amount of wall time = 492.654532
0: The maximum resident set size (KB) = 855504

Test thr PASS

OPERATION REQUIREMENT TEST WAS SUCCESSFUL
Mon Aug 18 16:43:55 UTC 2025
Elapsed time: 00h:53m:30s. Have a nice day!
Thu Aug 21 02:33:22 UTC 2025
Elapsed time: 00h:53m:07s. Have a nice day!
Loading