Skip to content

fixed restart non-reproducing issue 2588 when using MERRA2 climatology + Add output of ocean scalar variables for quick diagnostics #2632 + Fix CICD Jenkinsfile change default RT suite to full #2628 #2625

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jkbk2004 merged 31 commits into from
Mar 10, 2025
Merged

fixed restart non-reproducing issue 2588 when using MERRA2 climatology + Add output of ocean scalar variables for quick diagnostics #2632 + Fix CICD Jenkinsfile change default RT suite to full #2628 #2625

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
3d4b85b
establish mr2_rst
AnningCheng-NOAA Feb 27, 2025
9a679e1
changes in gitmodules
AnningCheng-NOAA Feb 27, 2025
c784902
tested control_p8 restart runs
AnningCheng-NOAA Feb 27, 2025
b1a8953
Add a few averaged variables to output, it is a subset of scalars at …
sanAkel Feb 27, 2025
04464d4
Create README.txt for CI Jenkins
BruceKropp-Raytheon Feb 27, 2025
18d5b98
Update Jenkinsfile fix default test suite to full
BruceKropp-Raytheon Feb 27, 2025
907f778
Merge branch 'develop' into fix/ci_jenkinsfile
BruceKropp-Raytheon Feb 27, 2025
195c46f
Merge branch 'ufs-community:develop' into feature/sanAkel/add-ocn-sca…
sanAkel Feb 28, 2025
4dca350
write 3 hourly global means instead of daily mean values because it i…
sanAkel Mar 4, 2025
9dc58f2
sync up fv3 feature branch hash
jkbk2004 Mar 5, 2025
dddf492
sync up against develop
jkbk2004 Mar 5, 2025
97c9354
add hera pre-test RT log
jkbk2004 Mar 5, 2025
d5cf6ea
Merge remote-tracking branch 'upstream/feature/sanAkel/add-ocn-scalar…
jkbk2004 Mar 5, 2025
44ed850
sync 3/5/2025 for mr2_rst
AnningCheng-NOAA Mar 6, 2025
fdc340b
merge to the develop branch in ufs-weather-model
AnningCheng-NOAA Mar 6, 2025
4b57225
Merge remote-tracking branch 'upstream/fix/ci_jenkinsfile' into mr2_rst
jkbk2004 Mar 6, 2025
a95bb6d
update fv3 feature branch hash
jkbk2004 Mar 6, 2025
6495c26
add hera RT log: passed
jkbk2004 Mar 7, 2025
42bb1ea
update test_changes.list
jkbk2004 Mar 7, 2025
4853c0f
add orion RT log: passed
jkbk2004 Mar 7, 2025
7877442
add hercules RT log: passed
jkbk2004 Mar 7, 2025
77f9a18
add gaeac5 RT log passed
FernandoAndrade-NOAA Mar 7, 2025
bb49bee
add gaeac6 RT log passed
FernandoAndrade-NOAA Mar 7, 2025
56366fd
add derecho RT log: passed
jkbk2004 Mar 7, 2025
32eb95f
ORT Jobs Completed.
epic-cicd-jenkins Mar 8, 2025
282e8bf
ORT Jobs Completed.
epic-cicd-jenkins Mar 8, 2025
9b8f912
ORT Jobs Completed.
epic-cicd-jenkins Mar 8, 2025
f2c739b
WCOSS2 RT Log: Passed
BrianCurtis-NOAA Mar 10, 2025
dcea345
Acorn RT Log: Passed
BrianCurtis-NOAA Mar 10, 2025
6d948e6
ORT Jobs Completed.
epic-cicd-jenkins Mar 10, 2025
c54b972
fv3 hash update and revert changes in gitmodules
jkbk2004 Mar 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .cicd/Jenkinsfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ def generateStage(nodeLabel) {
set +x

echo "Testing concluded...removing labels for $machine from $GIT_URL"
echo "https://api.github.com/repos/${GIT_OWNER}/${GIT_REPO_NAME}/issues/${CHANGE_ID}/labels /{$machine-RT,$machine-BL}"
echo "https://api.github.com/repos/${GIT_OWNER}/${GIT_REPO_NAME}/issues/${CHANGE_ID}/labels/{$machine-RT,$machine-BL}"
curl --silent -X DELETE -H "Accept: application/vnd.github.v3+json" -H "Authorization: Bearer ${GITHUB_TOKEN}" https://api.github.com/repos/${GIT_OWNER}/${GIT_REPO_NAME}/issues/${CHANGE_ID}/labels/{$machine-RT,$machine-BL}
'''
currentBuild.result = 'FAILURE'
Expand Down Expand Up @@ -131,7 +131,7 @@ def generateStage(nodeLabel) {
set +x

echo "Testing concluded...removing label ${label} for ${machine} from ${GIT_URL}"
echo "https://api.github.com/repos/${GIT_OWNER}/${GIT_REPO_NAME}/issues/${CHANGE_ID}/labels /${machine}-${label}"
echo "https://api.github.com/repos/${GIT_OWNER}/${GIT_REPO_NAME}/issues/${CHANGE_ID}/labels/${machine}-${label}"
curl --silent -X DELETE -H "Accept: application/vnd.github.v3+json" -H "Authorization: Bearer ${GITHUB_TOKEN}" https://api.github.com/repos/${GIT_OWNER}/${GIT_REPO_NAME}/issues/${CHANGE_ID}/labels/{$machine-RT,$machine-BL}
'''
}
Expand All @@ -150,7 +150,7 @@ pipeline {
parameters {
booleanParam name: 'WM_BUILD', defaultValue: false, description: 'Whether to attempt to compile the model code tests'
// Regression Test Suite ?
choice(name: 'WM_OPERATIONAL_TESTS', choices: ['default', 'control_p8', 'cpld_control_p8', 'comprehensive', 'rt.sh', 'none'], description: 'Specify the suite of tests to run')
choice(name: 'WM_OPERATIONAL_TESTS', choices: ['rt.conf', 'default', 'control_p8', 'cpld_control_p8', 'comprehensive', 'none'], description: 'Specify the suite of tests to run')
}
environment {
ACCNR = 'epic'
Expand Down
2 changes: 2 additions & 0 deletions .cicd/README.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
This folder contains scripts to perform automated builds.
It is intended to collect important performance metrics for trending.
2 changes: 1 addition & 1 deletion FV3
Open in desktop
Submodule FV3 updated from 41df0d to a46b8c
48 changes: 24 additions & 24 deletions tests/logs/OpnReqTests_control_p8_hera.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Tue Mar 4 13:59:06 UTC 2025
Mon Mar 10 14:49:46 UTC 2025
Start Operation Requirement Test


baseline dir = /scratch1/NCEPDEV/stmp4/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_bit_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_2432564/bit_base_bit_base
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_2514225/bit_base_bit_base
Checking test bit_base results ....
Moving baseline bit_base files ....
Moving sfcf000.nc .........OK
Expand Down Expand Up @@ -51,14 +51,14 @@ Moving baseline bit_base files ....
Moving RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Moving RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 264.470497
0: The maximum resident set size (KB) = 1452008
0: The total amount of wall time = 263.089636
0: The maximum resident set size (KB) = 1450812

Test bit_base PASS


baseline dir = /scratch1/NCEPDEV/stmp4/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_dbg_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_2432564/dbg_base_dbg_base
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_2514225/dbg_base_dbg_base
Checking test dbg_base results ....
Moving baseline dbg_base files ....
Moving sfcf000.nc .........OK
Expand Down Expand Up @@ -106,14 +106,14 @@ Moving baseline dbg_base files ....
Moving RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Moving RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 873.295769
0: The maximum resident set size (KB) = 1434860
0: The total amount of wall time = 894.435022
0: The maximum resident set size (KB) = 1424312

Test dbg_base PASS


baseline dir = /scratch1/NCEPDEV/stmp4/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_2432564/dcp_dcp
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_2514225/dcp_dcp
Checking test dcp results ....
Comparing sfcf000.nc .....USING NCCMP......OK
Comparing sfcf021.nc .....USING NCCMP......OK
Expand Down Expand Up @@ -160,14 +160,14 @@ Checking test dcp results ....
Comparing RESTART/20210323.060000.sfc_data.tile5.nc .....USING NCCMP......OK
Comparing RESTART/20210323.060000.sfc_data.tile6.nc .....USING NCCMP......OK

0: The total amount of wall time = 232.657622
0: The maximum resident set size (KB) = 1431144
0: The total amount of wall time = 255.195704
0: The maximum resident set size (KB) = 1439372

Test dcp PASS


baseline dir = /scratch1/NCEPDEV/stmp4/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_2432564/mpi_mpi
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_2514225/mpi_mpi
Checking test mpi results ....
Comparing sfcf000.nc .....USING NCCMP......OK
Comparing sfcf021.nc .....USING NCCMP......OK
Expand Down Expand Up @@ -214,14 +214,14 @@ Checking test mpi results ....
Comparing RESTART/20210323.060000.sfc_data.tile5.nc .....USING NCCMP......OK
Comparing RESTART/20210323.060000.sfc_data.tile6.nc .....USING NCCMP......OK

0: The total amount of wall time = 237.140431
0: The maximum resident set size (KB) = 1429124
0: The total amount of wall time = 240.708492
0: The maximum resident set size (KB) = 1431668

Test mpi PASS


baseline dir = /scratch1/NCEPDEV/stmp4/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_2432564/rst_rst
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_2514225/rst_rst
Checking test rst results ....
Comparing sfcf000.nc .....USING NCCMP......OK
Comparing sfcf021.nc .....USING NCCMP......OK
Expand Down Expand Up @@ -268,14 +268,14 @@ Checking test rst results ....
Comparing RESTART/20210323.060000.sfc_data.tile5.nc .....USING NCCMP......OK
Comparing RESTART/20210323.060000.sfc_data.tile6.nc .....USING NCCMP......OK

0: The total amount of wall time = 236.072984
0: The maximum resident set size (KB) = 1429740
0: The total amount of wall time = 236.404550
0: The maximum resident set size (KB) = 1413844

Test rst PASS


baseline dir = /scratch1/NCEPDEV/stmp4/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_2432564/std_base_std_base
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_2514225/std_base_std_base
Checking test std_base results ....
Moving baseline std_base files ....
Moving sfcf000.nc .........OK
Expand Down Expand Up @@ -323,14 +323,14 @@ Moving baseline std_base files ....
Moving RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Moving RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 240.588488
0: The maximum resident set size (KB) = 1427892
0: The total amount of wall time = 241.799857
0: The maximum resident set size (KB) = 1436660

Test std_base PASS


baseline dir = /scratch1/NCEPDEV/stmp4/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_2432564/thr_thr
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_2514225/thr_thr
Checking test thr results ....
Comparing sfcf000.nc .....USING NCCMP......OK
Comparing sfcf021.nc .....USING NCCMP......OK
Expand Down Expand Up @@ -377,11 +377,11 @@ Checking test thr results ....
Comparing RESTART/20210323.060000.sfc_data.tile5.nc .....USING NCCMP......OK
Comparing RESTART/20210323.060000.sfc_data.tile6.nc .....USING NCCMP......OK

0: The total amount of wall time = 236.799078
0: The maximum resident set size (KB) = 1420744
0: The total amount of wall time = 239.871402
0: The maximum resident set size (KB) = 1418840

Test thr PASS

OPERATION REQUIREMENT TEST WAS SUCCESSFUL
Tue Mar 4 15:08:51 UTC 2025
Elapsed time: 01h:09m:46s. Have a nice day!
Mon Mar 10 16:09:35 UTC 2025
Elapsed time: 01h:19m:49s. Have a nice day!
24 changes: 12 additions & 12 deletions tests/logs/OpnReqTests_cpld_control_nowave_noaero_p8_hera.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Tue Mar 4 12:17:24 UTC 2025
Mon Mar 10 13:17:06 UTC 2025
Start Operation Requirement Test


baseline dir = /scratch1/NCEPDEV/stmp4/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/cpld_control_c96_noaero_p8_dbg_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_1723361/dbg_base_dbg_base
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_2219240/dbg_base_dbg_base
Checking test dbg_base results ....
Moving baseline dbg_base files ....
Moving sfcf021.tile1.nc .........OK
Expand Down Expand Up @@ -66,14 +66,14 @@ Moving baseline dbg_base files ....
Moving RESTART/iced.2021-03-23-21600.nc .........OK
Moving RESTART/ufs.cpld.cpl.r.2021-03-23-21600.nc .........OK

0: The total amount of wall time = 2321.511982
0: The maximum resident set size (KB) = 1508268
0: The total amount of wall time = 2391.075563
0: The maximum resident set size (KB) = 1509636

Test dbg_base PASS


baseline dir = /scratch1/NCEPDEV/stmp4/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/cpld_control_c96_noaero_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_1723361/rst_rst
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_2219240/rst_rst
Checking test rst results ....
Comparing sfcf021.tile1.nc .....USING NCCMP......OK
Comparing sfcf021.tile2.nc .....USING NCCMP......OK
Expand Down Expand Up @@ -135,14 +135,14 @@ Checking test rst results ....
Comparing RESTART/iced.2021-03-23-21600.nc .....USING NCCMP......OK
Comparing RESTART/ufs.cpld.cpl.r.2021-03-23-21600.nc .....USING NCCMP......OK

0: The total amount of wall time = 885.357109
0: The maximum resident set size (KB) = 1500284
0: The total amount of wall time = 673.364321
0: The maximum resident set size (KB) = 1501156

Test rst PASS


baseline dir = /scratch1/NCEPDEV/stmp4/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/cpld_control_c96_noaero_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_1723361/std_base_std_base
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_2219240/std_base_std_base
Checking test std_base results ....
Moving baseline std_base files ....
Moving sfcf021.tile1.nc .........OK
Expand Down Expand Up @@ -205,11 +205,11 @@ Moving baseline std_base files ....
Moving RESTART/iced.2021-03-23-21600.nc .........OK
Moving RESTART/ufs.cpld.cpl.r.2021-03-23-21600.nc .........OK

0: The total amount of wall time = 906.585920
0: The maximum resident set size (KB) = 1515644
0: The total amount of wall time = 681.176217
0: The maximum resident set size (KB) = 1491940

Test std_base PASS

OPERATION REQUIREMENT TEST WAS SUCCESSFUL
Tue Mar 4 13:59:05 UTC 2025
Elapsed time: 01h:41m:41s. Have a nice day!
Mon Mar 10 14:49:46 UTC 2025
Elapsed time: 01h:32m:40s. Have a nice day!
30 changes: 15 additions & 15 deletions tests/logs/OpnReqTests_regional_control_hera.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Tue Mar 4 09:38:39 UTC 2025
Mon Mar 10 10:38:42 UTC 2025
Start Operation Requirement Test


baseline dir = /scratch1/NCEPDEV/stmp4/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/regional_control_bit_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_594890/bit_base_bit_base
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_1590323/bit_base_bit_base
Checking test bit_base results ....
Moving baseline bit_base files ....
Moving dynf000.nc .........OK
Expand All @@ -15,14 +15,14 @@ Moving baseline bit_base files ....
Moving NATLEV.GrbF00 .........OK
Moving NATLEV.GrbF06 .........OK

0: The total amount of wall time = 1398.696469
0: The maximum resident set size (KB) = 769184
0: The total amount of wall time = 1424.642932
0: The maximum resident set size (KB) = 755568

Test bit_base PASS


baseline dir = /scratch1/NCEPDEV/stmp4/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/regional_control_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_594890/dcp_dcp
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_1590323/dcp_dcp
Checking test dcp results ....
Comparing dynf000.nc .....USING NCCMP......OK
Comparing dynf006.nc .....USING NCCMP......OK
Expand All @@ -33,14 +33,14 @@ Checking test dcp results ....
Comparing NATLEV.GrbF00 .....USING CMP......OK
Comparing NATLEV.GrbF06 .....USING CMP......OK

0: The total amount of wall time = 2295.507380
0: The maximum resident set size (KB) = 708740
0: The total amount of wall time = 2290.668679
0: The maximum resident set size (KB) = 730512

Test dcp PASS


baseline dir = /scratch1/NCEPDEV/stmp4/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/regional_control_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_594890/std_base_std_base
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_1590323/std_base_std_base
Checking test std_base results ....
Moving baseline std_base files ....
Moving dynf000.nc .........OK
Expand All @@ -52,14 +52,14 @@ Moving baseline std_base files ....
Moving NATLEV.GrbF00 .........OK
Moving NATLEV.GrbF06 .........OK

0: The total amount of wall time = 2278.947058
0: The maximum resident set size (KB) = 719796
0: The total amount of wall time = 2295.886502
0: The maximum resident set size (KB) = 722460

Test std_base PASS


baseline dir = /scratch1/NCEPDEV/stmp4/role.epic/FV3_OPNREQ_TEST/OPNREQ_TEST/regional_control_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_594890/thr_thr
working dir = /scratch1/NCEPDEV/stmp2/role.epic/FV3_OPNREQ_TEST/opnReqTest_1590323/thr_thr
Checking test thr results ....
Comparing dynf000.nc .....USING NCCMP......OK
Comparing dynf006.nc .....USING NCCMP......OK
Expand All @@ -70,11 +70,11 @@ Checking test thr results ....
Comparing NATLEV.GrbF00 .....USING CMP......OK
Comparing NATLEV.GrbF06 .....USING CMP......OK

0: The total amount of wall time = 2279.672631
0: The maximum resident set size (KB) = 718664
0: The total amount of wall time = 2291.726981
0: The maximum resident set size (KB) = 729820

Test thr PASS

OPERATION REQUIREMENT TEST WAS SUCCESSFUL
Tue Mar 4 12:17:24 UTC 2025
Elapsed time: 02h:38m:45s. Have a nice day!
Mon Mar 10 13:17:06 UTC 2025
Elapsed time: 02h:38m:25s. Have a nice day!
Loading