Add master grib files to GFS HPSS archive for AIGFS#4203
Merged
Conversation
junwang-noaa
previously approved these changes
Oct 31, 2025
Contributor
|
@CatherineThomas-NOAA I got some feedback on AIGFSv1 that users ask for more frequent output (GFS is hourly, and AIGFS is 6 hourly). May I ask if we can save hourly master file from f000-f006, or at aleast f003 so that we may have 3 hourly data for training? Thanks |
Contributor
Author
|
I ran the CI on Gaea C6. The lowres v17 test has the following new entries in the gdas.tar file: @junwang-noaa - Is this sufficient? |
junwang-noaa
approved these changes
Nov 3, 2025
Contributor
Author
|
Thanks @junwang-noaa. Taking out of draft. |
Contributor
Author
|
CI Tests on Gaea-C6 pass: |
aerorahul
approved these changes
Nov 3, 2025
Contributor
aerorahul
left a comment
aerorahul
left a comment
There was a problem hiding this comment.
looks good. Just one question.
Comment on lines
+177
to
+178
| - "{{ COMIN_ATMOS_MASTER | relpath(ROTDIR) }}/{{ head }}master.f003.grib2" | ||
| - "{{ COMIN_ATMOS_MASTER | relpath(ROTDIR) }}/{{ head }}master.f006.grib2" |
Contributor
There was a problem hiding this comment.
Should we include the idx files?
Contributor
There was a problem hiding this comment.
Actually, I wonder if we are even generating the index files for these. Probably are not.
Contributor
Author
There was a problem hiding this comment.
I had thought about this as well. We're at least not saving them to COM, we don't have them in our parallels. I didn't check if there is a silent failure somewhere though.
@WenMeng-NOAA - Can you confirm whether the master grib files should have idx files saved in COM as well?
Contributor
There was a problem hiding this comment.
The idx files for the master are not created, similar to the sflux files. The sflux idx files are generated in the atmos_products task. I'll start generating the master.idx files if the group wants that, and start including it in COM. The master files are strictly internal to GFS (until now for archiving for AIGFS).
| {% endfor %} | ||
|
|
||
|
|
||
| # Forecast Master GRIB2 data for F006 |
Contributor
There was a problem hiding this comment.
Suggested change
| # Forecast Master GRIB2 data for F006 | |
| # Forecast Master GRIB2 data for F003 and F006 |
weihuang-jedi
added a commit
to NOAA-EPIC/global-workflow-cloud
that referenced
this pull request
Nov 5, 2025
…NOAA-EPIC/global-workflow-cloud into feature/use_container_spack-stack-1.9.2 * 'feature/use_container_spack-stack-1.9.2' of github.com:NOAA-EPIC/global-workflow-cloud: reverse few changes re-sync with EMC repo Add master grib files to GFS HPSS archive for AIGFS (NOAA-EMC#4203) Update Snow filenames to comply with EE2 (NOAA-EMC#4195) Rename files for JEDI atm EE2 (NOAA-EMC#4193) Generate `pres_b` files for `RUN=gdas` and update `APCP` to `598`. (NOAA-EMC#4196) Update checks for MOM6 restarts when performing a re-run on failure (NOAA-EMC#4179) Decrease HPSS storage for GFS retros and address hpss bugs (NOAA-EMC#4184) Add noaacloud to ufsda case in dev/ush/load_modules (NOAA-EMC#4198) Remove replay from global workflow (NOAA-EMC#4182) Add IODA stats text file to COM (NOAA-EMC#4176) Update UFS_UTILS submodule (NOAA-EMC#4178) Atm COM File Rename to Standardized Form (NOAA-EMC#4117) Replace cp with cpfs/cpreq for atomic copies to COM directories (NOAA-EMC#4130) Create a UPP module for the global workflow (NOAA-EMC#4174) Refactor marine DA tasks (NOAA-EMC#4160) Delay ocean post-processing trigger to next-next forecast (NOAA-EMC#4167) Make options hashes Remove multiple option from static data template Fix static_data yaml (descriptions and labels) Fix static_data yaml (remove colon) Add Ursa to and remove C5 from list of HPCs in the bug report template (NOAA-EMC#4164) Rename marine (ocean/ice) files following EE2 conventions (NOAA-EMC#4162) Add attributes to Gaussian grid sfcanl file (NOAA-EMC#4149) Remove the snow analysis from archive (NOAA-EMC#4157) Update verif-global to fix pcp failures on special cases (NOAA-EMC#4154) Add CRTM fix directory paths to global-workflow (NOAA-EMC#4143) Update UFS Model (NOAA-EMC#4138) Add functionality to assimilate the new snow observations (NOAA-EMC#4132)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
AIGFS currently trains with 0.25 degree grib2 data. For future development, higher resolution data will be used.
The files that are needed from GDAS are the F006 and analysis master grib files (edit to add: F003 as well). This PR updates the HPSS archive list to archive those additional grib files and their associated idx files for the deterministic gdas forecast only.
Resolves #4191
Type of change
Change characteristics
How has this been tested?
CI on Gaea-C6 passed. HPSS tarballs were checked for relevant files.
Checklist