Update checks for MOM6 restarts when performing a re-run on failure by aerorahul · Pull Request #4179 · NOAA-EMC/global-workflow

aerorahul · 2025-10-24T20:46:45Z

Description

This PR:

Resolves Restart runs are not ensuring MOM6 output has completed writing. #3822
employs additional checks on MOM6 state when restarting upon re-run for robustness. This check is skipped when FHOUT_OCN > 6 such as in the SFS runs where FHOUT_OCN=24. This causes problems noted in the issue comment here

This PR also:

has a minor fix in ush/preamble.sh that prints the Begin date time in human readable format similar to End date time message.

Type of change

Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO

How has this been tested?

Tests 1, 2, 3 below have FHMAX_GFS=384 and restart_interval_gfs=48

Ran C48_S2SW case forecast out to FHMAX_GFS without stopping
Start running C48_S2SW forecast
a. kill forecast job at fhr=126. Restarts are available at hrs 48, 96
b. re-submit forecast job. The job picks the restarts at hr 96 and completes
Start running C48_S2SW forecast
a. kill forecast job at fhr = 102. Restarts are available at hrs 48, 96
b. re-submit forecast job. The job picks restarts at hr 48 (ocean forecast history file at 105 is not available, rendering restart at hr 96 invalid) and completes.
Ran C48_S2SW test start to end.
Ran C48_S2SWA_gefs test to test the segment functionality.

Checklist

JessicaMeixner-NOAA · 2025-10-27T12:27:53Z

+      else
+        # Also check for MOM6 history file availability
+        # TODO: Need to adapt for SFS where averaging period may be different, need to generalize
+        hdate=$(date -u -d "${rdate:0:8} ${rdate:8:2}:00:00 + 3 hours" +"%Y_%m_%d_%H")  # MOM6 history is averaged 6 hrs


Should this be generalized now with FHOUT_OCN ?

Also - This should have some sort of if not gdas/enkfgdas because the output is different for those runs. Should those runs have different checks here?

Generalized to use FHOUT_OCN. I had committed what I had on Friday before leaving work, and a TODO to remind me about it when I pick it up on Monday.
2f1650f addresses the first part of this comment.

RERUN is not available for RUN = gdas|enkfgdas as their forecast lengths are less than 12 hours (15 if you count IAU).
https://github.com/aerorahul/global-workflow/blob/2f1650f2eb7237ab95373e50cb081097545781d0/ush/forecast_det.sh#L38-L42
enkfgfs does not have a forecast segment.
I'll test this for a RUN=gefs with segments to ensure this change does not break that.

…uman-readable as end

aerorahul · 2025-10-28T13:56:33Z

GEFS test passed.
Merged develop after renaming PR.
Opening up for review.

JessicaMeixner-NOAA

This resolves the open issue.

emcbot · 2025-10-28T19:00:55Z

C96mx100_S2S FAILED on Gaeac6 (pipeline ID: 5409)

In directory: /gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4179_87175e4c_5409/RUNTESTS/EXPDIR/C96mx100_S2S_87175e4c-5409

Error Log Files:


/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4179_87175e4c_5409/RUNTESTS/COMROOT/C96mx100_S2S_87175e4c-5409/logs/1994050100/sfs_fcst_mem000_seg1.log
/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4179_87175e4c_5409/RUNTESTS/COMROOT/C96mx100_S2S_87175e4c-5409/logs/1994050100/sfs_fcst_mem001_seg1.log
/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4179_87175e4c_5409/RUNTESTS/COMROOT/C96mx100_S2S_87175e4c-5409/logs/1994050100/sfs_fcst_mem002_seg1.log

View Error Logs: (sfs_fcst_mem000_seg1.log) (sfs_fcst_mem001_seg1.log) (sfs_fcst_mem002_seg1.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

aerorahul · 2025-10-28T19:22:14Z

I did not test the SFS case in my testing. I will do that and report back. SFS has a restart_interval_gfs=12 and does 24hr averaging of the ocean output. The checks for restarts are probably either not enough or too long.

aerorahul · 2025-10-29T14:34:34Z

See comment in the issue #3822 and advise if the approach there is acceptable.

…s AND ocean output averaged files. Deal with this another time

DavidHuber-NOAA

Looks good. Just a nitpick on variable name.

DavidHuber-NOAA · 2025-10-30T15:44:57Z

Launching tests on C6.

DavidHuber-NOAA · 2025-10-30T21:06:52Z

All tests passed on C6. Merging.

…NOAA-EPIC/global-workflow-cloud into feature/use_container_spack-stack-1.9.2 * 'feature/use_container_spack-stack-1.9.2' of github.com:NOAA-EPIC/global-workflow-cloud: reverse few changes re-sync with EMC repo Add master grib files to GFS HPSS archive for AIGFS (NOAA-EMC#4203) Update Snow filenames to comply with EE2 (NOAA-EMC#4195) Rename files for JEDI atm EE2 (NOAA-EMC#4193) Generate `pres_b` files for `RUN=gdas` and update `APCP` to `598`. (NOAA-EMC#4196) Update checks for MOM6 restarts when performing a re-run on failure (NOAA-EMC#4179) Decrease HPSS storage for GFS retros and address hpss bugs (NOAA-EMC#4184) Add noaacloud to ufsda case in dev/ush/load_modules (NOAA-EMC#4198) Remove replay from global workflow (NOAA-EMC#4182) Add IODA stats text file to COM (NOAA-EMC#4176) Update UFS_UTILS submodule (NOAA-EMC#4178) Atm COM File Rename to Standardized Form (NOAA-EMC#4117) Replace cp with cpfs/cpreq for atomic copies to COM directories (NOAA-EMC#4130) Create a UPP module for the global workflow (NOAA-EMC#4174) Refactor marine DA tasks (NOAA-EMC#4160) Delay ocean post-processing trigger to next-next forecast (NOAA-EMC#4167) Make options hashes Remove multiple option from static data template Fix static_data yaml (descriptions and labels) Fix static_data yaml (remove colon) Add Ursa to and remove C5 from list of HPCs in the bug report template (NOAA-EMC#4164) Rename marine (ocean/ice) files following EE2 conventions (NOAA-EMC#4162) Add attributes to Gaussian grid sfcanl file (NOAA-EMC#4149) Remove the snow analysis from archive (NOAA-EMC#4157) Update verif-global to fix pcp failures on special cases (NOAA-EMC#4154) Add CRTM fix directory paths to global-workflow (NOAA-EMC#4143) Update UFS Model (NOAA-EMC#4138) Add functionality to assimilate the new snow observations (NOAA-EMC#4132)

update checks for MOM6 restarts when performing a re-run on failure

937f90e

aerorahul requested review from DavidHuber-NOAA, JessicaMeixner-NOAA and jiandewang October 24, 2025 20:47

JessicaMeixner-NOAA reviewed Oct 27, 2025

View reviewed changes

aerorahul added 3 commits October 27, 2025 17:33

use FHOUT_OCN instead of 6. minor fix in preamble to write Begin in h…

2f1650f

…uman-readable as end

Merge branch 'develop' into feature/issue-3822b

2363e6a

Merge branch 'develop' into feature/issue-3822b

fd45bed

aerorahul marked this pull request as ready for review October 28, 2025 13:56

JessicaMeixner-NOAA previously approved these changes Oct 28, 2025

View reviewed changes

jiandewang previously approved these changes Oct 28, 2025

View reviewed changes

emcbot added CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 and removed CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 labels Oct 28, 2025

DavidHuber-NOAA previously approved these changes Oct 28, 2025

View reviewed changes

emcbot added CI-Gaeac6-Running and removed CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 labels Oct 28, 2025

emcbot added CI-Gaeac6-Failed **Bot use only** CI testing on Gaea C6 for this PR has failed and removed CI-Gaeac6-Running labels Oct 28, 2025

aerorahul mentioned this pull request Oct 29, 2025

Restart runs are not ensuring MOM6 output has completed writing. #3822

Closed

aerorahul added 2 commits October 29, 2025 11:59

Merge branch 'develop' into feature/issue-3822b

5b00e49

SFS runs with FHOUT_OCN=24 and poses issues with checking the restart…

86d40bb

…s AND ocean output averaged files. Deal with this another time

aerorahul dismissed stale reviews from jiandewang, JessicaMeixner-NOAA, and DavidHuber-NOAA via 86d40bb October 30, 2025 15:25

aerorahul requested a review from jiandewang October 30, 2025 15:26

aerorahul requested review from DavidHuber-NOAA and JessicaMeixner-NOAA October 30, 2025 15:26

aerorahul removed the CI-Gaeac6-Failed **Bot use only** CI testing on Gaea C6 for this PR has failed label Oct 30, 2025

DavidHuber-NOAA reviewed Oct 30, 2025

View reviewed changes

Comment thread ush/forecast_det.sh

DavidHuber-NOAA reviewed Oct 30, 2025

View reviewed changes

Comment thread ush/forecast_det.sh

DavidHuber-NOAA approved these changes Oct 30, 2025

View reviewed changes

aerorahul mentioned this pull request Oct 30, 2025

Enable checks for MOM6 ocean output in cases where FHOUT_OCN > 6 (e.g. SFS) #4199

Open

Merge branch 'develop' into feature/issue-3822b

a56154f

emcbot added CI-Gaeac6-Passed **Bot use only** CI testing on Gaea C6 for this PR has completed successfully and removed CI-Gaeac6-Running labels Oct 30, 2025

DavidHuber-NOAA merged commit 5f0cfee into NOAA-EMC:develop Oct 30, 2025
5 checks passed

aerorahul mentioned this pull request Oct 31, 2025

UFS will not write more than three hundred restart files #2761

Closed

aerorahul deleted the feature/issue-3822b branch October 31, 2025 17:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update checks for MOM6 restarts when performing a re-run on failure#4179

Update checks for MOM6 restarts when performing a re-run on failure#4179
DavidHuber-NOAA merged 7 commits into
NOAA-EMC:developfrom
aerorahul:feature/issue-3822b

aerorahul commented Oct 24, 2025 •

edited

Loading

Uh oh!

JessicaMeixner-NOAA Oct 27, 2025

Uh oh!

aerorahul Oct 27, 2025

Uh oh!

aerorahul commented Oct 28, 2025

Uh oh!

JessicaMeixner-NOAA left a comment

Uh oh!

emcbot commented Oct 28, 2025

Uh oh!

aerorahul commented Oct 28, 2025

Uh oh!

aerorahul commented Oct 29, 2025

Uh oh!

Uh oh!

Uh oh!

DavidHuber-NOAA left a comment

Uh oh!

DavidHuber-NOAA commented Oct 30, 2025

Uh oh!

DavidHuber-NOAA commented Oct 30, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

aerorahul commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

JessicaMeixner-NOAA Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

aerorahul Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

aerorahul commented Oct 28, 2025

Uh oh!

JessicaMeixner-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

emcbot commented Oct 28, 2025

Uh oh!

aerorahul commented Oct 28, 2025

Uh oh!

aerorahul commented Oct 29, 2025

Uh oh!

Uh oh!

Uh oh!

DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

DavidHuber-NOAA commented Oct 30, 2025

Uh oh!

DavidHuber-NOAA commented Oct 30, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

aerorahul commented Oct 24, 2025 •

edited

Loading