update to MOM6 main 20240729 commit (gfdl-to-main-2024-05-31) by jiandewang · Pull Request #2381 · ufs-community/ufs-weather-model

jiandewang · 2024-07-31T14:15:55Z

Commit Queue Requirements:

Fill out all sections of this template.
All sub component pull requests have been reviewed by their code managers.
Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
Commit 'test_changes.list' from previous step

Description:

MOM6 main repo. merged the "gfdl-to-main-2024-05-31" on 20240729 (see detail at mom-ocean/MOM6#1631). Need to pull it into dev/emc correspondingly.
Note this PR will require a new BL because

a new variable "h_ML" is being added in the restart file (see detail at mom-ocean/MOM6@7305528)

a bug fixing on "incorrect PPM:H3 tracer advection stencil" (see detail at NOAA-EMC/MOM6@3fac1c5). Note we can retain original answer by setting USE_HUYNH_STENCIL_BUG=True.

we will also set DEFAULT_ANSWER_DATE=20231231 in 5x5 setting as currently it is using its default value which is too aggressive (suggested by GFDL)

Commit Message:

      add MOM_dynamics_split_RK2b.F90 and MOM_EOS_base_type.F90 in cmake list
      add extra 16PE for cpld_debug_pdlib job
      set DEFAULT_ANSWER_DATE=20231231 in 5x5 case
      using MOM6 restart file as BL for sfs and hafs_mom jobs
      switching FIX_USTAR_GUSTLESS_BUG to USTAR_GUSTLESS_BUG in MOM_input
      USE_HUYNH_STENCIL_BUG=True (false will have mpi jobs fail to match their control)

Please provide concise information for The UFS-WM and/or each sub-component:

MOM6 -

Priority:

High: Time-sensitive project.
High: Reason: this PR has been lasted for 2 months to reach final stage and a new MOM6 PR has arrived. Need to finish this PR before I can do any clean test for the new PR.

Git Tracking

UFSWM:

Closes update to MOM6 main 20240729 commit (gfdl-to-main-2024-05-31) #2380

Sub component Pull Requests:

MOM6: update to main 20240729 commit (gfdl-to-main-2024-05-31) NOAA-EMC/MOM6#136

UFSWM Blocking Dependencies:

None

Changes

Regression Test Changes (Please commit test_changes.list):

PR Updates/Changes Baselines.

Input data Changes:

None.

Library Changes/Upgrades:

No Updates

Testing Log:

add MOM_dynamics_split_RK2b.F90 and MOM_EOS_base_type.F90 in cmake list add extra 8PE for cpld_debug_pdlib job set DEFAULT_ANSWER_DATE=20231231 in 5x5 case add USE_HUYNH_STENCIL_BUG = True in MOM_input using MOM6 restart file as BL for sfs and hafs_mom jobs

switch FIX_USTAR_GUSTLESS_BUG=F to USTAR_GUSTLESS_BUG=T

jkbk2004 · 2024-08-12T23:46:44Z

@jiandewang can you sync up branch?

jiandewang · 2024-08-13T01:43:15Z

@jkbk2004 just sync-ed my PR branch

jiandewang · 2024-08-13T03:06:28Z

note for hafs_regional_storm_following_1nest_atm_ocn_wav_mom6 and cpld_control_sfs, you need to create RESTART directory inside BL dir and copy MOMresnc to it.

jiandewang · 2024-08-13T22:30:45Z

are we running this PR ? I don't see any email traffic on it today.

zach1221 · 2024-08-14T00:01:07Z

are we running this PR ? I don't see any email traffic on it today.

Yes, tests are running.

on-behalf-of @ufs-community <ecc.platform@noaa.gov>

FernandoAndrade-NOAA · 2024-08-14T16:01:06Z

Leaving a note that Jet will likely have to be skipped. There is a system issue preventing git clones, I've let the system admins know so they can take a look.

FernandoAndrade-NOAA · 2024-08-15T03:37:53Z

Apologies for the delayed Hera and Gaea logs. RTs were started, but I am unable to log onto the machines to check up on the results due to the token login issues today.

FernandoAndrade-NOAA · 2024-08-16T15:53:22Z

@DusanJovic-NOAA Are we skipping Acorn for this PR?

DusanJovic-NOAA · 2024-08-16T15:54:58Z

@DusanJovic-NOAA Are we skipping Acorn for this PR?

Yes.

jkbk2004 · 2024-08-16T16:04:46Z

Hera is very busy. I see a long job queue on Hera. Jet lfs4 issue continues. If we need to recover anything, we need to follow on those tow machine with next pr.

FernandoAndrade-NOAA · 2024-08-16T18:01:28Z

Looks like we'll be skipping hera and jet due to system issues this PR then, we'll catch up on the next PR @jkbk2004

DeniseWorthen · 2024-08-16T18:33:04Z

@FernandoAndrade-NOAA The next PR is currently scheduled to be my warmstart PR. If you plan on skipping baseline creation on Hera, then the next PR must create all baselines---mine only requires one.

I'm also not clear on what the hurry is. It is late on Friday afternoon. Are you planning to start a new PR today?

zach1221 · 2024-08-16T18:41:43Z

@DeniseWorthen I think this PR is listed as high priority above, so we were trying to get it merged before weekend.

DeniseWorthen · 2024-08-16T18:42:14Z

And anyone running RTs this weekend on hera will fail.

jiandewang · 2024-08-16T18:56:04Z

MOM6 merged (hash # 00f8ea20), submodule reverted

jiandewang · 2024-08-16T18:59:50Z

And anyone running RTs this weekend on hera will fail.

hafs_regional_storm_following_1nest_atm_ocn_wav_mom6 will fail on HERA

jiandewang · 2024-08-16T19:06:01Z

And anyone running RTs this weekend on hera will fail.

hafs_regional_storm_following_1nest_atm_ocn_wav_mom6 will fail on HERA

I just checked HERA BL /scratch2/NAGAPE/epic/UFS-WM_RT/NEMSfv3gfs/develop-20240813/hafs_regional_storm_following_1nest_atm_ocn_wav_mom6_intel/RESTART

-rw-r--r-- 1 role.epic epic 4197540137 Aug 16 13:41 20200825.150000.MOM.res.nc
-rw-r--r-- 1 role.epic epic 2276048626 Aug 16 13:41 20200825.150000.MOM.res_1.nc

these two were generated ~1.5hr ago so they will be fine

jiandewang · 2024-08-17T22:25:20Z

@jkbk2004 @FernandoAndrade-NOAA my test indicates" BL for hafs_regional_storm_following_1nest_atm_ocn_wav_mom6_intel" has not been updated on HERA. This job will fail if people run latest UWM. It will show "20200825.150000.MOM.res.nc" not identical.

jiandewang · 2024-08-17T22:41:44Z

And anyone running RTs this weekend on hera will fail.

hafs_regional_storm_following_1nest_atm_ocn_wav_mom6 will fail on HERA

I just checked HERA BL /scratch2/NAGAPE/epic/UFS-WM_RT/NEMSfv3gfs/develop-20240813/hafs_regional_storm_following_1nest_atm_ocn_wav_mom6_intel/RESTART

-rw-r--r-- 1 role.epic epic 4197540137 Aug 16 13:41 20200825.150000.MOM.res.nc -rw-r--r-- 1 role.epic epic 2276048626 Aug 16 13:41 20200825.150000.MOM.res_1.nc

these two were generated ~1.5hr ago so they will be fine

the above is what I checked Friday around 4pm, they are correct one (I compared them with my own run and got b4b). but now (Sat. 6:40pm) I got
-rw-r--r-- 1 role.epic epic 4197540137 Aug 16 20:21 20200825.150000.MOM.res.nc
-rw-r--r-- 1 role.epic epic 2276048626 Aug 16 20:21 20200825.150000.MOM.res_1.nc
from the time stamp of file being generated, apparently there is updating on them and the new one is not correct

jkbk2004 · 2024-08-17T22:50:18Z

@jiandewang let me check with develop branch

jiandewang · 2024-08-17T22:53:22Z

I compared these two yesterday
/scratch2/NAGAPE/epic/UFS-WM_RT/NEMSfv3gfs/develop-20240813_/hafs_regional_storm_following_1nest_atm_ocn_wav_mom6_intel/RESTART

/scratch2/NAGAPE/epic/UFS-WM_RT/NEMSfv3gfs/develop-20240813/hafs_regional_storm_following_1nest_atm_ocn_wav_mom6_intel/RESTART

they are not the same, and the BL directory "develop-20240813" contained the correct MOM restart file. But right now when I compare them, they are the same. So somehow files in "develop-20240813_" was being copied to develop-20240813" in the past half day which shouldn't.

jiandewang · 2024-08-17T22:54:58Z

@jiandewang let me check with develop branch

@jkbk2004 a clean way to do is re-run this job and replace the MOM restart file in BL dir.

jkbk2004 · 2024-08-18T00:38:48Z

@jiandewang let me check with develop branch

@jkbk2004 a clean way to do is re-run this job and replace the MOM restart file in BL dir.

@jiandewang recovered ok.

…r-model into develop * 'develop' of https://github.com/ufs-community/ufs-weather-model: update mom6 to its main repo. 20240824 commit (FMA) (ufs-community#2412) Update CMEPS; fix aux history functionality for float variable type; Switch to using Aux history files in atm_ds2s_docn_dice test; Remove IFI tests (was ufs-community#2417) (ufs-community#2395) Combination for CCPP-physics ufs-community#213 and ufs-community#218 (H2O scheme refactor and C3/SAS/MYNN fix) (ufs-community#2408) Unify CDEPS gfs, cfsr, and gefs datm datamodes + Improve error checking in rt.sh (2388) + Add ability to read increment files on native cubed sphere grid (2304) (ufs-community#2389) sync with head of NOAA-EMC UPP develop (ufs-community#2326) Allow use of downscaled warmstart files for cpld_control_sfs test (ufs-community#2375) update to MOM6 main 20240729 commit (gfdl-to-main-2024-05-31) (ufs-community#2381)

jiandewang and others added 7 commits July 29, 2024 23:13

using MOM6 test branch

fd911a0

add MOM_dynamics_split_RK2b.F90 and MOM_EOS_base_type.F90 in cmake list add extra 8PE for cpld_debug_pdlib job set DEFAULT_ANSWER_DATE=20231231 in 5x5 case add USE_HUYNH_STENCIL_BUG = True in MOM_input using MOM6 restart file as BL for sfs and hafs_mom jobs

Merge branch 'ufs-community:develop' into feature/update-MOM6-20240531

d322aa8

remove USE_HUYNH_STENCIL_BUG = True in MOM_input

e511aa2

switch FIX_USTAR_GUSTLESS_BUG=F to USTAR_GUSTLESS_BUG=T

commit test_changes.list

97adf10

remove USE_HUYNH_STENCIL_BUG from HAFS-mom6

9267080

update MOM6 branch

670e53b

using own PR test_changes.list

76828cc

using own test_changes.list

a4b4321

switch FIX_USTAR_GUSTLESS_BUG to USTAR_GUSTLESS_BUG for MOM_input_hafs

e29f4ef

zach1221 added Baseline Updates Current baselines will be updated. Ready for Commit Queue The PR is ready for the Commit Queue. All checkboxes in PR template have been checked. labels Aug 13, 2024

Update bl_date.conf

2e6b0bf

zach1221 added jenkins-ort run ORT testing and removed jenkins-ort run ORT testing labels Aug 13, 2024

zach1221 and others added 2 commits August 13, 2024 19:06

add orion RT logs: passed

877f1c5

ORT Jobs Completed.

ca7f5fc

on-behalf-of @ufs-community <ecc.platform@noaa.gov>

DeniseWorthen previously approved these changes Aug 14, 2024

View reviewed changes

add hercules RT logs: passed

82d262a

zach1221 dismissed DeniseWorthen’s stale review via 82d262a August 14, 2024 12:45

add derecho RT logs: passed

a47ca68

add gaea RT logs: passed

12418d0

DusanJovic-NOAA and others added 5 commits August 16, 2024 13:14

add wcoss2 RT logs: passed

c587137

rerun gaea RT log pass

12eb230

add orion RT logs: passed

f9e0775

add hercules RT logs: passed

f9c9232

add derecho RT logs: passed

3c4aa28

FernandoAndrade-NOAA mentioned this pull request Aug 16, 2024

update to main 20240729 commit (gfdl-to-main-2024-05-31) NOAA-EMC/MOM6#136

Merged

revert MOM6 submodule

eb4e537

zach1221 requested review from FernandoAndrade-NOAA and jkbk2004 August 16, 2024 19:10

jkbk2004 approved these changes Aug 16, 2024

View reviewed changes

FernandoAndrade-NOAA approved these changes Aug 16, 2024

View reviewed changes

jkbk2004 merged commit 6d0454c into ufs-community:develop Aug 17, 2024

jiandewang deleted the feature/update-MOM6-20240531 branch April 2, 2025 01:59

Conversation

jiandewang commented Jul 31, 2024 • edited by zach1221 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Commit Queue Requirements:

Description:

Commit Message:

Priority:

Git Tracking

UFSWM:

Sub component Pull Requests:

UFSWM Blocking Dependencies:

Changes

Regression Test Changes (Please commit test_changes.list):

Input data Changes:

Library Changes/Upgrades:

Testing Log:

Uh oh!

jkbk2004 commented Aug 12, 2024

Uh oh!

jiandewang commented Aug 13, 2024

Uh oh!

jiandewang commented Aug 13, 2024

Uh oh!

jiandewang commented Aug 13, 2024

Uh oh!

zach1221 commented Aug 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FernandoAndrade-NOAA commented Aug 14, 2024

Uh oh!

FernandoAndrade-NOAA commented Aug 15, 2024

Uh oh!

FernandoAndrade-NOAA commented Aug 16, 2024

Uh oh!

DusanJovic-NOAA commented Aug 16, 2024

Uh oh!

jkbk2004 commented Aug 16, 2024

Uh oh!

FernandoAndrade-NOAA commented Aug 16, 2024

Uh oh!

DeniseWorthen commented Aug 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zach1221 commented Aug 16, 2024

Uh oh!

DeniseWorthen commented Aug 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jiandewang commented Aug 16, 2024

Uh oh!

jiandewang commented Aug 16, 2024

Uh oh!

jiandewang commented Aug 16, 2024

Uh oh!

jiandewang commented Aug 17, 2024

Uh oh!

jiandewang commented Aug 17, 2024

Uh oh!

jkbk2004 commented Aug 17, 2024

Uh oh!

jiandewang commented Aug 17, 2024

Uh oh!

jiandewang commented Aug 17, 2024

Uh oh!

jkbk2004 commented Aug 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

jiandewang commented Jul 31, 2024 •

edited by zach1221

Loading

zach1221 commented Aug 14, 2024 •

edited

Loading

DeniseWorthen commented Aug 16, 2024 •

edited

Loading

DeniseWorthen commented Aug 16, 2024 •

edited

Loading