Skip to content

Reorganize parm/gdas#4011

Merged
DavidHuber-NOAA merged 51 commits into
NOAA-EMC:developfrom
DavidNew-NOAA:feature/parm-clean
Sep 9, 2025
Merged

Reorganize parm/gdas#4011
DavidHuber-NOAA merged 51 commits into
NOAA-EMC:developfrom
DavidNew-NOAA:feature/parm-clean

Conversation

@DavidNew-NOAA
Copy link
Copy Markdown
Contributor

@DavidNew-NOAA DavidNew-NOAA commented Sep 4, 2025

Description

This PR aims to finally organize the parm/gdas directory, which currently has files distributed between Global Workflow and GDASApp, since many of its directories are symlinked from the parm directory of GDASApp, and move all files into GDASApp.

The files will be renamed in GDASApp companion PR *** to give them a unified naming convention. This PR then gives the environment variables that point to these files a unified naming convention that corresponds directly to that file name. This will make it easier for developers to find the underlying file that these variables point to, a common issue when working with JEDI tasks.

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? YES

How has this been tested?

Clone, build, and full CI suite on Hera

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

@DavidNew-NOAA DavidNew-NOAA changed the title Feature/parm clean Reorganize parm/gdas Sep 4, 2025
@DavidNew-NOAA DavidNew-NOAA self-assigned this Sep 4, 2025
@DavidNew-NOAA DavidNew-NOAA linked an issue Sep 4, 2025 that may be closed by this pull request
@DavidNew-NOAA DavidNew-NOAA added the needs submodule update Requires submodule PRs to be merged label Sep 4, 2025
@DavidNew-NOAA DavidNew-NOAA added the CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera label Sep 8, 2025
@DavidNew-NOAA
Copy link
Copy Markdown
Contributor Author

DavidNew-NOAA commented Sep 8, 2025

@DavidHuber-NOAA I added CI-Hera-Running label to indicate that I'm running CI on Hera manually

@DavidNew-NOAA DavidNew-NOAA added CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Sep 8, 2025
@emcbot emcbot added CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 CI-Gaeac6-Running and removed CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 labels Sep 8, 2025
@DavidNew-NOAA DavidNew-NOAA removed the Ready for CI This PR is ready for CM's to launch automated testing on the platforms label Sep 8, 2025
@DavidNew-NOAA
Copy link
Copy Markdown
Contributor Author

Hera CI passes with hash cbe9eca

******** C48_ATM ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Sep 08 2025 14:30:14    Sep 08 2025 15:24:04
******** C48mx500_3DVarAOWCDA ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Sep 08 2025 14:30:15    Sep 08 2025 14:42:07
202103250000        Done    Sep 08 2025 14:30:15    Sep 08 2025 16:02:04
******** C48mx500_hybAOWCDA ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Sep 08 2025 14:30:14    Sep 08 2025 14:48:04
202103250000        Done    Sep 08 2025 14:30:14    Sep 08 2025 15:32:03
******** C48_S2SWA_gefs ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Sep 08 2025 14:30:15    Sep 08 2025 15:30:14
******** C48_S2SW ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Sep 08 2025 14:30:15    Sep 08 2025 15:26:05
******** C96_atm3DVar ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Sep 08 2025 14:30:14    Sep 08 2025 14:48:04
202112210000        Done    Sep 08 2025 14:30:14    Sep 08 2025 16:52:05
202112210600        Done    Sep 08 2025 14:30:14    Sep 08 2025 16:28:03
******** C96C48_hybatmDA ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Sep 08 2025 14:30:14    Sep 08 2025 14:48:04
202112210000        Done    Sep 08 2025 14:30:14    Sep 08 2025 16:48:04
202112210600        Done    Sep 08 2025 14:30:14    Sep 08 2025 16:14:03
******** C96C48_hybatmsnowDA ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Sep 08 2025 14:30:15    Sep 08 2025 14:48:04
202112201800        Done    Sep 08 2025 14:30:15    Sep 08 2025 16:46:04
202112210000        Done    Sep 08 2025 14:30:15    Sep 08 2025 16:12:04
******** C96C48_hybatmsoilDA ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202205150600        Done    Sep 08 2025 14:30:15    Sep 08 2025 14:48:04
202205151200        Done    Sep 08 2025 14:30:15    Sep 08 2025 16:50:12
202205151800        Done    Sep 08 2025 14:30:15    Sep 08 2025 16:32:03
******** C96C48mx500_S2SW_cyc_gfs ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Sep 08 2025 14:30:15    Sep 08 2025 14:48:05
202112201800        Done    Sep 08 2025 14:30:15    Sep 08 2025 16:46:04
202112210000        Done    Sep 08 2025 14:30:15    Sep 08 2025 17:30:14
202112211800        Done    Sep 08 2025 14:50:11    Sep 08 2025 17:32:04
******** C96C48_ufs_hybatmDA ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    Sep 08 2025 14:30:14    Sep 08 2025 14:48:04
202402240000        Done    Sep 08 2025 14:30:14    Sep 08 2025 17:06:04
202402240600        Done    Sep 08 2025 14:30:14    Sep 08 2025 16:30:13
******** C96_gcafs_cycled_noDA ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Sep 08 2025 14:30:14    Sep 08 2025 14:48:04
202112201800        Done    Sep 08 2025 14:30:14    Sep 08 2025 16:42:04
202112210000        Done    Sep 08 2025 14:30:14    Sep 08 2025 15:24:04
******** C96_gcafs_cycled ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Sep 08 2025 14:30:14    Sep 08 2025 14:48:04
202112201800        Done    Sep 08 2025 14:30:14    Sep 08 2025 16:44:04
202112210000        Done    Sep 08 2025 14:30:14    Sep 08 2025 15:32:03
******** C96mx100_S2S ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
199405010000        Done    Sep 08 2025 14:30:14    Sep 08 2025 16:04:04

@DavidNew-NOAA DavidNew-NOAA added CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Sep 8, 2025
@emcbot emcbot added CI-Gaeac6-Passed **Bot use only** CI testing on Gaea C6 for this PR has completed successfully and removed CI-Gaeac6-Running labels Sep 8, 2025
@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor

WCOSS2 g-w CI

Install DavidNew-NOAA:feature/parm-clean at cbe9eca on Cactus. Run g-w CI with the following results

/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_ATM_pr4011
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Sep 08 2025 14:17:15    Sep 08 2025 16:15:50
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_3DVarAOWCDA_pr4011
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Sep 08 2025 14:17:17    Sep 08 2025 14:36:23
202103250000        Done    Sep 08 2025 14:17:17    Sep 08 2025 16:37:41
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_hybAOWCDA_pr4011
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Sep 08 2025 14:17:19    Sep 08 2025 14:36:32
202103250000        Done    Sep 08 2025 14:17:19    Sep 08 2025 16:26:03
 
SKIP C48_S2SWA_gefs_RT on wcoss2
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_S2SWA_gefs_pr4011
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Sep 08 2025 14:17:21    Sep 08 2025 15:36:06
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_S2SW_extended_pr4011
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Sep 08 2025 14:17:24    Sep 08 2025 17:11:35
202103231800        Done    Sep 08 2025 14:17:24    Sep 08 2025 17:20:28
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_S2SW_pr4011
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Sep 08 2025 14:18:01    Sep 08 2025 16:16:20
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96_atm3DVar_extended_pr4011
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Sep 08 2025 14:18:04    Sep 08 2025 14:47:40
202112210000      Active    Sep 08 2025 14:18:04             -          
202112210600      Active    Sep 08 2025 14:18:04             -          
202112211200      Active    Sep 08 2025 14:48:05             -          
202112211800    Inactive             -                       -          
 
SKIP C96_atm3DVar on wcoss2
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmDA_pr4011
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Sep 08 2025 14:18:06    Sep 08 2025 14:47:55
202112210000        Done    Sep 08 2025 14:18:06    Sep 08 2025 17:20:43
202112210600        Done    Sep 08 2025 14:18:06    Sep 08 2025 16:51:56
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmsnowDA_pr4011
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Sep 08 2025 14:18:08    Sep 08 2025 14:48:00
202112201800        Done    Sep 08 2025 14:18:08    Sep 08 2025 17:12:12
202112210000        Done    Sep 08 2025 14:18:08    Sep 08 2025 16:52:11
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmsoilDA_pr4011
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202205150600        Done    Sep 08 2025 14:18:10    Sep 08 2025 14:48:04
202205151200        Done    Sep 08 2025 14:18:10    Sep 08 2025 17:17:06
202205151800        Done    Sep 08 2025 14:18:10    Sep 08 2025 17:36:36
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48mx500_S2SW_cyc_gfs_pr4011
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Sep 08 2025 14:18:56    Sep 08 2025 14:48:11
202112201800        Done    Sep 08 2025 14:18:56    Sep 08 2025 17:17:11
202112210000        Done    Sep 08 2025 14:18:56    Sep 08 2025 18:26:30
202112211800        Done    Sep 08 2025 14:48:20    Sep 08 2025 18:22:13
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_ufs_hybatmDA_pr4011
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    Sep 08 2025 14:19:08    Sep 08 2025 14:48:15
202402240000        Done    Sep 08 2025 14:19:08    Sep 08 2025 17:42:00
202402240600        Done    Sep 08 2025 14:19:08    Sep 08 2025 17:26:54
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96_gcafs_cycled_noDA_pr4011
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Sep 08 2025 14:19:09    Sep 08 2025 14:48:19
202112201800        Done    Sep 08 2025 14:19:09    Sep 08 2025 17:02:11
202112210000        Done    Sep 08 2025 14:19:09    Sep 08 2025 16:28:10
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96_gcafs_cycled_pr4011
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Sep 08 2025 14:19:12    Sep 08 2025 14:48:23
202112201800        Done    Sep 08 2025 14:19:12    Sep 08 2025 17:13:05
202112210000        Done    Sep 08 2025 14:19:12    Sep 08 2025 16:55:19
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96mx100_S2S_pr4011
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
199405010000        Done    Sep 08 2025 14:19:14    Sep 08 2025 15:57:24

All g-w CI cases configured to run on WCOSS2 successfully completed. The only exception is C96_atm3DVar_extended which is still running. Thus far, no errors have occurred in C96_atm3DVar_extended. The status of the WCOSS2 C96_atm3DVar_extended will be reported once all jobs complete.

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor

WCOSS2 g-w CI

C96_atm3DVar_extended finished. All jobs in all cycles successfully ran to completion.

/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96_atm3DVar_extended_pr4011
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Sep 08 2025 14:18:04    Sep 08 2025 14:47:40
202112210000        Done    Sep 08 2025 14:18:04    Sep 08 2025 20:11:15
202112210600        Done    Sep 08 2025 14:18:04    Sep 08 2025 20:51:36
202112211200        Done    Sep 08 2025 14:48:05    Sep 08 2025 23:00:41
202112211800        Done    Sep 08 2025 20:14:08    Sep 09 2025 01:21:12

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor

@DavidHuber-NOAA : The CI-Wcoss2-Running label can be changed to CI-Wcoss2-Passed. I would do so but I don't know g-w CM policy. Hence this comment.

@RussTreadon-NOAA RussTreadon-NOAA self-requested a review September 9, 2025 01:41
Copy link
Copy Markdown
Contributor

@RussTreadon-NOAA RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

g-w CI Passed on Gaea C6, Hera and WCOSS2 (Cactus).

Approve.

Copy link
Copy Markdown
Contributor

@guillaumevernieres guillaumevernieres left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@DavidHuber-NOAA DavidHuber-NOAA added CI-Wcoss2-Passed CI testing on WCOSS for this PR has completed successfully and removed CI-Wcoss2-Running CI testing on WCOSS for this PR is in-progress labels Sep 9, 2025
@DavidHuber-NOAA
Copy link
Copy Markdown
Contributor

All tests passed. Merging.

@DavidHuber-NOAA DavidHuber-NOAA merged commit 29e5e44 into NOAA-EMC:develop Sep 9, 2025
5 checks passed
@DavidNew-NOAA
Copy link
Copy Markdown
Contributor Author

Thanks @RussTreadon-NOAA snd @DavidHuber-NOAA for testing

aerorahul pushed a commit to aerorahul/global-workflow that referenced this pull request Sep 10, 2025
This aims to finally organize the `parm/gdas` directory, which
currently has files distributed between Global Workflow and GDASApp,
since many of its directories are symlinked from the `parm` directory of
GDASApp, and move all files into GDASApp.

The files will be renamed in the GDASApp to give them a
unified naming convention. This then gives the environment variables
that point to these files a unified naming convention that corresponds
directly to that file name. This will make it easier for developers to
find the underlying file that these variables point to, a common issue
when working with JEDI tasks.
DavidNew-NOAA added a commit to NOAA-EMC/GDASApp that referenced this pull request Jan 16, 2026
# Description

This PR does a couple things.

1. It moves every file that lives in Global Workflow `parm/gdas` into
GDASApp `parm` and into the appropriate subdirectory (`parm/atm`,
`parm/aero`, etc)
2. All `parm` files that feed into the wxflow FileHandler or configure
the GW `Jedi` class (most of the files in `parm`) are given a unified
naming convention of `(domain)_(task)_(stage/save)_(thing being staged
or saved).yaml.j2` or `(domain)_(task)_jedi_config.yaml.j2`
respectively. For example, the file that stages atmospheric
deterministic backgrounds is called `atm_det_stage_bkg.yaml.j2`.
3. `parm/soca` becomes `parm/marine` for consistency.
4. Anything that doesn't seem to be used in `parm` is moved to
`parm/obsolete`.

# Companion PRs

NOAA-EMC/global-workflow#4011

# Issues

Refs NOAA-EMC/global-workflow#3973

---------

Co-authored-by: root <root>
Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>
Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>
Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>
Co-authored-by: RussTreadon-NOAA <26926959+RussTreadon-NOAA@users.noreply.github.com>
Co-authored-by: danholdaway <danholdaway@users.noreply.github.com>
Co-authored-by: Dan Holdaway <27729500+danholdaway@users.noreply.github.com>
Co-authored-by: Walter Kolczynski - NOAA <Walter.Kolczynski@noaa.gov>
Co-authored-by: RussTreadon-NOAA <russ.treadon@noaa.gov>
Co-authored-by: Guillaume Vernieres <guillaume.vernieres@noaa.gov>
Co-authored-by: AntonMFernando-NOAA <167725623+AntonMFernando-NOAA@users.noreply.github.com>
Co-authored-by: Anil Kumar <108816337+AnilKumar-NOAA@users.noreply.github.com>
Co-authored-by: TerrenceMcGuinness-NOAA <terrence.mcguinness@cox.net>
Co-authored-by: Andrew.Tangborn <Andrew.Tangborn@noaa.gov>
Co-authored-by: Jiarui Dong <Jiarui.Dong@noaa.gov>
Co-authored-by: Cory Martin <Cory.R.Martin@gaea68.ncrc.gov>
Co-authored-by: Travis Elless <113720457+TravisElless-NOAA@users.noreply.github.com>
Co-authored-by: Anna Shlyaeva <anna.v.shlyaeva@noaa.gov>
Co-authored-by: Kate Friedman <kate.friedman@noaa.gov>
Co-authored-by: Cory Martin <Cory.R.Martin@gaea67.ncrc.gov>
Co-authored-by: Cory Martin <Cory.R.Martin@gaea66.ncrc.gov>
Co-authored-by: Kevin Dougherty <kevin.dougherty@noaa.gov>
Co-authored-by: Kevin Dougherty <69815622+kevindougherty-noaa@users.noreply.github.com>
Co-authored-by: Anna Shlyaeva <annash@ucar.edu>
Co-authored-by: yuanxue2870 <136842224+yuanxue2870@users.noreply.github.com>
Co-authored-by: Yaping Wang <49168260+ypwang19@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI-Gaeac6-Passed **Bot use only** CI testing on Gaea C6 for this PR has completed successfully CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Wcoss2-Passed CI testing on WCOSS for this PR has completed successfully

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Move parm/gdas files to parm in GDASApp

7 participants