Update the marine analysis archiving by JohnSteffen-NOAA · Pull Request #4082 · NOAA-EMC/global-workflow

JohnSteffen-NOAA · 2025-09-18T21:02:25Z

Description

This PR updates the GDAS marine analysis archiving to include the post-processed sea ice increments and it changes the suffix to ".nc" to match the filename extension. Additionally, a new GFS marine analysis archiving yaml is added when the GFS analysis/forecast is run.

Resolves #4020

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model
- wxflow

How has this been tested?

The GDAS/GFS marine analysis archiving has been tested on WCOSS2.

Checklist

…essed ice increment file and includes a new gfsocean analysis archiving capability. Resolves NOAA-EMC#4020

…during arch_tars

JohnSteffen-NOAA · 2025-09-25T19:38:49Z

I was able to test the GDAS/GFS archiving changes to the marine analysis on WCOSS2 using a fresh build of my g-w fork. The archiving works as intended (mostly). The postproc ice increment file is now included and the gfsocean_analysis task is included in the rocoto workflow. The experiment directory can be found here:

/lfs/h2/emc/da/noscrub/john.steffen/expts/test-archive/

And the HPSS archive location is here:
/NCEPDEV/emc-global/1year/john.steffen/WCOSS2/scratch/test-archive/

However, the new gfs_arch_tar_gfsocean_analysis task does technically fail at the end of the job due the backup_tarballs.yaml file still being modified when the archiving job begins. See the relevant section of the log (./COMROOT/test-archive/logs/2025070100/gfs_arch_tar_gfsocean_analysis.log),

HTAR Create complete for /NCEPDEV/emc-global/1year/john.steffen/WCOSS2/scratch/test-archive/2025070100/gfsocean_analysis.tar. 12736800768 bytes written for 43 member files, max threads: 4 Transfer time: 86.469 seconds (147.300 MB/s) wallclock/user/sys: 87.737 11.086 49.241 seconds
HTAR: HTAR SUCCESSFUL
[connecting to hpsscore1.fairmont.rdhpcs.noaa.gov/1217]
ERROR: htar_LFW_Thread: (line 2679,lfwthread 3) file gfs.20250701/00/conf/backup_tarballs.yaml:size changed from 150 to 123
###WARNING  htar returned non-zero exit status.

Recommendations on how to correct this error is appreciated!

DavidHuber-NOAA

Looks good. One minor suggestion.

DavidHuber-NOAA · 2025-09-26T17:55:11Z

I would like to test this on Hera or C6. However, C6 is not ready to use yet following its OS upgrade and Hera is maxed out at the moment. I will schedule CI testing for this PR for Monday.

EricSinsky-NOAA · 2025-09-26T20:31:09Z


            if self.options['do_ocean']:
-                tarball_types.extend(['ocean_6hravg', 'ocean_grib2', 'gfs_flux_1p00'])
+                tarball_types.extend(['ocean_6hravg', 'ocean_grib2', 'gfs_flux_1p00', 'gfsocean_analysis'])


Thank you for adding gfsocean analysis to the archive, @JohnSteffen-NOAA. I was also working on adding the gfs ocean analysis to the archive since these are needed to initialize GEFS. In my testing, when I added gfsocean_analysis here (line 2249), I found that some CI tests failed when mode is forecast-only because these files are not generated in this mode. I suggest adding gfsocean_analysis only when mode is cycled (please see commit 9177934).

Good catch! Thanks @EricSinsky-NOAA.

Suggested change

tarball_types.extend(['ocean_6hravg', 'ocean_grib2', 'gfs_flux_1p00', 'gfsocean_analysis'])

tarball_types.extend(['ocean_6hravg', 'ocean_grib2', 'gfs_flux_1p00'])

if self.options['do_jediocnvar'] and self.app_config.mode == 'cycled':

tarball_types.append('gfsocean_analysis')

@DavidHuber-NOAA , done. I'm testing now on WCOSS2.

@DavidHuber-NOAA I've corrected the error from ci_pytest and this PR is now ready for CI testing on Hera/Gaea. Thanks!

… turned on and cycled

DavidHuber-NOAA · 2025-09-30T18:17:28Z

Running CI on Hera.

DavidHuber-NOAA · 2025-10-01T13:30:45Z

All tests passed on Hera. Merging.

…into feature/adjust_tasks_per_node_layout * 'develop' of github.com:NOAA-EPIC/global-workflow-cloud: Update CODEOWNERS for new oceanice_products assignments (NOAA-EMC#4134) Improve GW forecast timing of high-resolution runs with findings from containerized GW (NOAA-EMC#4123) Update CODEOWNERS to change job assignments (NOAA-EMC#4128) Update j-jobs to be bash linter compliant (NOAA-EMC#4124) Update the marine analysis archiving (NOAA-EMC#4082) Port tracker/genesis tasks to cloud; load wgrib2; add noaacloud to load_gw_gsi_modules (NOAA-EMC#4111) Only run wave bnd pnt for gfs (NOAA-EMC#4112)

…NOAA-EPIC/global-workflow-cloud into feature/use_container_spack-stack-1.9.2 * 'feature/use_container_spack-stack-1.9.2' of github.com:NOAA-EPIC/global-workflow-cloud: only double quota varaibles only double quota varaibles need run_mpmd.sh changes remove config.resources.URSA from sfs recover ursa.yaml testing on ursa working on aws remove unused dir Update the marine analysis archiving (NOAA-EMC#4082) Port tracker/genesis tasks to cloud; load wgrib2; add noaacloud to load_gw_gsi_modules (NOAA-EMC#4111) Only run wave bnd pnt for gfs (NOAA-EMC#4112)

This updates the gdasocean analysis archiving to inclue the post-proc…

1ad0c24

…essed ice increment file and includes a new gfsocean analysis archiving capability. Resolves NOAA-EMC#4020

JohnSteffen-NOAA requested a review from aerorahul as a code owner September 18, 2025 21:02

JohnSteffen-NOAA marked this pull request as draft September 18, 2025 21:02

JohnSteffen-NOAA marked this pull request as ready for review September 18, 2025 21:03

JohnSteffen-NOAA marked this pull request as draft September 18, 2025 21:04

JohnSteffen-NOAA and others added 3 commits September 23, 2025 13:21

Merge branch 'develop' into feature/marine_analysis_arch

c5a61d8

This update adds the gfsocean_analysis archiving task to be included …

5d64fc0

…during arch_tars

Fix ci_pytest failure

7f88bd1

JohnSteffen-NOAA marked this pull request as ready for review September 25, 2025 19:39

JohnSteffen-NOAA requested a review from DavidHuber-NOAA as a code owner September 25, 2025 19:39

DavidHuber-NOAA previously approved these changes Sep 26, 2025

View reviewed changes

Comment thread dev/workflow/rocoto/gfs_tasks.py Outdated

Code clean up to include the gfsocean_analysis archiving

6ae18e0

JohnSteffen-NOAA dismissed DavidHuber-NOAA’s stale review via 6ae18e0 September 26, 2025 19:56

EricSinsky-NOAA reviewed Sep 26, 2025

View reviewed changes

john steffen and others added 3 commits September 29, 2025 13:39

Edits to include the gfs ocean analysis archives only if marine DA is…

1c74f8d

… turned on and cycled

Fix pytest error for do_jediocnvar

92593ad

Merge branch 'develop' into feature/marine_analysis_arch

ae6b08e

DavidHuber-NOAA approved these changes Sep 30, 2025

View reviewed changes

DavidHuber-NOAA merged commit cfe3215 into NOAA-EMC:develop Oct 1, 2025
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update the marine analysis archiving#4082

Update the marine analysis archiving#4082
DavidHuber-NOAA merged 8 commits into
NOAA-EMC:developfrom
JohnSteffen-NOAA:feature/marine_analysis_arch

JohnSteffen-NOAA commented Sep 18, 2025 •

edited

Loading

Uh oh!

JohnSteffen-NOAA commented Sep 25, 2025 •

edited

Loading

Uh oh!

DavidHuber-NOAA left a comment

Uh oh!

Uh oh!

DavidHuber-NOAA commented Sep 26, 2025

Uh oh!

EricSinsky-NOAA Sep 26, 2025

Uh oh!

DavidHuber-NOAA Sep 29, 2025

Uh oh!

DavidHuber-NOAA Sep 29, 2025 •

edited

Loading

Uh oh!

JohnSteffen-NOAA Sep 29, 2025

Uh oh!

JohnSteffen-NOAA Sep 30, 2025

Uh oh!

DavidHuber-NOAA commented Sep 30, 2025

Uh oh!

DavidHuber-NOAA commented Oct 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

JohnSteffen-NOAA commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

JohnSteffen-NOAA commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

DavidHuber-NOAA commented Sep 26, 2025

Uh oh!

EricSinsky-NOAA Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

DavidHuber-NOAA Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

DavidHuber-NOAA Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JohnSteffen-NOAA Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

JohnSteffen-NOAA Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

DavidHuber-NOAA commented Sep 30, 2025

Uh oh!

DavidHuber-NOAA commented Oct 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

JohnSteffen-NOAA commented Sep 18, 2025 •

edited

Loading

JohnSteffen-NOAA commented Sep 25, 2025 •

edited

Loading

DavidHuber-NOAA Sep 29, 2025 •

edited

Loading