Decrease HPSS storage for GFS retros and address hpss bugs#4184
Merged
DavidHuber-NOAA merged 47 commits intoOct 30, 2025
Conversation
to reduce archive list
removes the *yaml glob used in archiving and replace with explicit filesnames
Conflicts: parm/archive/enkf_marine_grp.yaml.j2 parm/archive/gdasice.yaml.j2 parm/archive/gdasocean.yaml.j2 parm/archive/gdasocean_analysis.yaml.j2
Contributor
Author
|
Gaea CI: I also ran on WCOSS2 because I forgot HPSS doesn't run there. I can post those logs in the morning. HPSS location is: /NCEPDEV/emc-global/1year/Jessica.Meixner/GAEAC6/scratch/C*_t01pr4184 |
Contributor
Author
|
An additional experiment that uses the code here (plus other changes) can be found at: /NCEPDEV/emc-global/1year/Jessica.Meixner/GAEAC6/scratch/rt18t01 WCOSS2 CI tests also passed: |
aerorahul
previously approved these changes
Oct 30, 2025
Contributor
aerorahul
left a comment
aerorahul
left a comment
There was a problem hiding this comment.
all my comments have been addressed.
thanks.
Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>
DavidHuber-NOAA
approved these changes
Oct 30, 2025
Contributor
DavidHuber-NOAA
left a comment
DavidHuber-NOAA
left a comment
There was a problem hiding this comment.
Thanks @JessicaMeixner-NOAA
Contributor
|
Launching CI on Hera. |
emcbot
added
the
CI-Hera-Ready
DavidHuber-NOAA
added
CI-Gaeac6-Passed (cm)
emcbot
added
CI-Hera-Building
|
C96C48_ufs_hybatmDA FAILED on Hera (pipeline ID: 5442) In directory: Error Log Files: View Error Logs: (gdas_analcalc_fv3jedi.log) (gfs_analcalc_fv3jedi.log) This failure was detected automatically by global-workflow's CI/CD Pipeline |
emcbot
added
CI-Hera-Failed
Contributor
This test should not have run on Hera (and I'm not sure how it did). I will monitor the remaining experiments. Removing the failed label. |
DavidHuber-NOAA
added
CI-Hera-Running (CM)
CatherineThomas-NOAA
approved these changes
Oct 30, 2025
Contributor
CatherineThomas-NOAA
left a comment
CatherineThomas-NOAA
left a comment
There was a problem hiding this comment.
The code changes and HPSS archives look good to me.
DavidHuber-NOAA
added
CI-Hera-Passed (cm)
Contributor
|
All tests passed on Hera. Merging. |
22 tasks
weihuang-jedi
added a commit
to NOAA-EPIC/global-workflow-cloud
that referenced
this pull request
Nov 5, 2025
…NOAA-EPIC/global-workflow-cloud into feature/use_container_spack-stack-1.9.2 * 'feature/use_container_spack-stack-1.9.2' of github.com:NOAA-EPIC/global-workflow-cloud: reverse few changes re-sync with EMC repo Add master grib files to GFS HPSS archive for AIGFS (NOAA-EMC#4203) Update Snow filenames to comply with EE2 (NOAA-EMC#4195) Rename files for JEDI atm EE2 (NOAA-EMC#4193) Generate `pres_b` files for `RUN=gdas` and update `APCP` to `598`. (NOAA-EMC#4196) Update checks for MOM6 restarts when performing a re-run on failure (NOAA-EMC#4179) Decrease HPSS storage for GFS retros and address hpss bugs (NOAA-EMC#4184) Add noaacloud to ufsda case in dev/ush/load_modules (NOAA-EMC#4198) Remove replay from global workflow (NOAA-EMC#4182) Add IODA stats text file to COM (NOAA-EMC#4176) Update UFS_UTILS submodule (NOAA-EMC#4178) Atm COM File Rename to Standardized Form (NOAA-EMC#4117) Replace cp with cpfs/cpreq for atomic copies to COM directories (NOAA-EMC#4130) Create a UPP module for the global workflow (NOAA-EMC#4174) Refactor marine DA tasks (NOAA-EMC#4160) Delay ocean post-processing trigger to next-next forecast (NOAA-EMC#4167) Make options hashes Remove multiple option from static data template Fix static_data yaml (descriptions and labels) Fix static_data yaml (remove colon) Add Ursa to and remove C5 from list of HPCs in the bug report template (NOAA-EMC#4164) Rename marine (ocean/ice) files following EE2 conventions (NOAA-EMC#4162) Add attributes to Gaussian grid sfcanl file (NOAA-EMC#4149) Remove the snow analysis from archive (NOAA-EMC#4157) Update verif-global to fix pcp failures on special cases (NOAA-EMC#4154) Add CRTM fix directory paths to global-workflow (NOAA-EMC#4143) Update UFS Model (NOAA-EMC#4138) Add functionality to assimilate the new snow observations (NOAA-EMC#4132)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR combines updates from PR #4159 which
Fixes #4158
Removes archiving of log files that have "tar" in the name so all tar-archiving log files will be skipped and avoid an error that log file is still being written.
Reduces wave archiving #4150
And updates from @CatherineThomas-NOAA based on discussions with the marine-DA team to reduce archiving.
It also updated ufs-utils and ufs-weather-model to the top of develop when I started testing.
Fixes #4158
Fixes #4155
Fixes #4150
Type of change
Change characteristics
How has this been tested?
A full generate workflows has been run on gaea c6. I will run another set now on wcoss2 after merging in multiple commits from develop. Wanted to get feedback before launching that.
Checklist