Skip to content

Refactor marine DA tasks#4160

Merged
DavidNew-NOAA merged 43 commits into
NOAA-EMC:developfrom
DavidNew-NOAA:feature/marine-refactor
Oct 25, 2025
Merged

Refactor marine DA tasks#4160
DavidNew-NOAA merged 43 commits into
NOAA-EMC:developfrom
DavidNew-NOAA:feature/marine-refactor

Conversation

@DavidNew-NOAA
Copy link
Copy Markdown
Contributor

@DavidNew-NOAA DavidNew-NOAA commented Oct 20, 2025

Description

This PR makes a few changes, refactoring marine DA tasks:

  1. It consolidates all parm YAML files required for marine DA tasks into single master files for each task.
  2. It changes marine DA tasks to now inherit from the Python Analysis class. This class consolidates variables and methods universal to all JEDI DA tasks.
  3. It refactors the marine LETKF task to use the JCB and the Python Jedi class.
  4. It turns marine LETKF back on as the default of JEDI hybrid marine DA.
  5. It adds an marineanlletkf (required but missed until now) dependency to the earc_tars task.

Resolves #4161

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? YES
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? YES (If YES, please add a link to any PRs that are pending.)

How has this been tested?

Clone and build on Hera
Run all CI tests

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

@DavidNew-NOAA DavidNew-NOAA marked this pull request as ready for review October 20, 2025 20:22
@DavidNew-NOAA DavidNew-NOAA added the blocked Issue is currently being blocked by another issue. Include blocking issue # in description label Oct 20, 2025
@DavidNew-NOAA
Copy link
Copy Markdown
Contributor Author

This PR and its companions are on hold until Rahul's https://github.com/NOAA-EMC/GDASApp/tree/feature/marine_rename branch is merged

@DavidNew-NOAA
Copy link
Copy Markdown
Contributor Author

@DavidHuber-NOAA I've resolved any remaining issues with @aerorahul and this PR is ready for CI

@emcbot emcbot added CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 CI-Ursa-Ready **CM use only** PR is ready for CI testing on Ursa CI-Ursa-Building **Bot use only** CI testing is cloning/building on Ursa CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 and removed CI-Ursa-Ready **CM use only** PR is ready for CI testing on Ursa CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 labels Oct 24, 2025
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA removed CI-Gaeac6-Failed **Bot use only** CI testing on Gaea C6 for this PR has failed CI-Ursa-Failed **Bot use only** CI testing on Ursa for this PR has failed labels Oct 24, 2025
Comment on lines +35 to +36
SOCA_INPUT_FIX_DIR: {{ HOMEgfs }}/fix/gdas/soca/1440x1080x75/soca
SOCA_ANL_GEOM: {{ HOMEgfs }}/fix/gdas/soca/720x540x75/soca
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are these not in config.base if they are used across so many jobs?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I have another PR planned at some point to remove utilities from ush/python/pygfs/utils, so I will put them in config.base there.

@emcbot emcbot added CI-Ursa-Running **Bot use only** CI testing on Ursa for this PR is in-progress CI-Gaeac6-Running CI-Ursa-Passed **Bot use only** CI testing on Ursa for this PR has completed successfully CI-Gaeac6-Passed **Bot use only** CI testing on Gaea C6 for this PR has completed successfully and removed CI-Ursa-Building **Bot use only** CI testing is cloning/building on Ursa CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 CI-Ursa-Running **Bot use only** CI testing on Ursa for this PR is in-progress CI-Gaeac6-Running labels Oct 24, 2025
@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor

WCOSS2 g-w CI

Install DavidNew-NOAA:feature/marine-refactor at 078f15a on WCOSS2 (Cactus). Run g-w CI with the following results.

SKIP C48_ATM_ecflow on wcoss2
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_ATM_pr4160
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Oct 24 2025 18:21:08    Oct 24 2025 19:25:53
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_3DVarAOWCDA_pr4160
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Oct 24 2025 18:21:10    Oct 24 2025 18:35:42
202103250000        Done    Oct 24 2025 18:21:10    Oct 24 2025 20:15:54
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_hybAOWCDA_pr4160
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Oct 24 2025 18:21:13    Oct 24 2025 18:35:46
202103250000        Done    Oct 24 2025 18:21:13    Oct 24 2025 19:55:59
 
SKIP C48_S2SWA_gefs_RT on wcoss2
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_S2SWA_gefs_pr4160
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Oct 24 2025 18:21:15    Oct 24 2025 19:51:14
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_S2SW_extended_pr4160
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Oct 24 2025 18:21:18    Oct 24 2025 21:11:14
202103231800        Done    Oct 24 2025 18:21:18    Oct 24 2025 21:21:15
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_S2SW_pr4160
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Oct 24 2025 18:21:20    Oct 24 2025 19:26:23
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96_atm3DVar_extended_pr4160
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Oct 24 2025 18:21:23    Oct 24 2025 18:41:33
202112210000        Done    Oct 24 2025 18:21:23    Oct 24 2025 23:26:18
202112210600        Done    Oct 24 2025 18:21:23    Oct 25 2025 00:36:05
202112211200      Active    Oct 24 2025 18:46:31             -          
202112211800      Active    Oct 24 2025 23:36:00             -          
 
SKIP C96_atm3DVar on wcoss2
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmDA_pr4160
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Oct 24 2025 18:21:25    Oct 24 2025 18:41:37
202112210000        Done    Oct 24 2025 18:21:25    Oct 24 2025 21:16:33
202112210600        Done    Oct 24 2025 18:21:25    Oct 24 2025 21:21:27
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmsnowDA_pr4160
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Oct 24 2025 18:21:27    Oct 24 2025 18:41:41
202112201800        Done    Oct 24 2025 18:21:27    Oct 24 2025 21:11:39
202112210000        Done    Oct 24 2025 18:21:27    Oct 24 2025 21:21:31
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmsoilDA_pr4160
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202205150600        Done    Oct 24 2025 18:21:29    Oct 24 2025 18:41:44
202205151200        Done    Oct 24 2025 18:21:29    Oct 24 2025 21:21:35
202205151800        Done    Oct 24 2025 18:21:29    Oct 24 2025 21:21:35
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48mx500_S2SW_cyc_gfs_pr4160
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Oct 24 2025 18:21:31    Oct 24 2025 18:41:48
202112201800        Done    Oct 24 2025 18:21:31    Oct 24 2025 21:51:33
202112210000        Done    Oct 24 2025 18:21:31    Oct 24 2025 22:41:55
202112211800        Done    Oct 24 2025 18:46:44    Oct 24 2025 22:41:55
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_ufs_hybatmDA_pr4160
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    Oct 24 2025 18:21:34    Oct 24 2025 18:41:52
202402240000        Done    Oct 24 2025 18:21:34    Oct 24 2025 21:41:46
202402240600        Done    Oct 24 2025 18:21:34    Oct 24 2025 21:46:26
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96_gcafs_cycled_noDA_pr4160
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Oct 24 2025 18:21:35    Oct 24 2025 18:41:55
202112201800        Done    Oct 24 2025 18:21:35    Oct 24 2025 20:52:08
202112210000        Done    Oct 24 2025 18:21:35    Oct 24 2025 19:52:07
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96_gcafs_cycled_pr4160
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Oct 24 2025 18:21:38    Oct 24 2025 18:41:59
202112201800        Done    Oct 24 2025 18:21:38    Oct 24 2025 20:56:07
202112210000        Done    Oct 24 2025 18:21:38    Oct 24 2025 20:21:44
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96mx100_S2S_pr4160
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
199405010000        Done    Oct 24 2025 18:21:41    Oct 24 2025 20:16:39

All jobs in all cases run on Cactus successfully completed all jobs with the exception of g-w CI case C96_atm3DVar_extended. The 20211220 18Z, 20211221 00Z, and 20211221 06Z cycles of this case completed. The 12Z and 18Z cycles are still running. No errors, thus far, have occurred in the C96_atm3DVar_extended case.

@DavidNew-NOAA
Copy link
Copy Markdown
Contributor Author

Thanks @RussTreadon-NOAA for the Cactus testing. Ursa and C6 pass as well. Merging

@DavidNew-NOAA DavidNew-NOAA merged commit 46c4013 into NOAA-EMC:develop Oct 25, 2025
5 checks passed
@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor

WCOSS2 g-w CI

For the sake of consistency, document the fact that this PR successfully completed C96_atm3DVar_extended on Cactus.

/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96_atm3DVar_extended_pr4160
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Oct 24 2025 18:21:23    Oct 24 2025 18:41:33
202112210000        Done    Oct 24 2025 18:21:23    Oct 24 2025 23:26:18
202112210600        Done    Oct 24 2025 18:21:23    Oct 25 2025 00:36:05
202112211200        Done    Oct 24 2025 18:46:31    Oct 25 2025 02:11:21
202112211800        Done    Oct 24 2025 23:36:00    Oct 25 2025 05:05:53

Thank you @DavidNew-NOAA for merging this PR in g-w develop. This simplifies testing of GDASApp PR #1951. Hera stable-nightly testing has been moved back to g-w develop. Feel free to delete DavidNew-NOAA:feature/marine-refactor at your convenience.

@DavidNew-NOAA DavidNew-NOAA deleted the feature/marine-refactor branch October 25, 2025 11:23
weihuang-jedi added a commit to NOAA-EPIC/global-workflow-cloud that referenced this pull request Nov 5, 2025
…NOAA-EPIC/global-workflow-cloud into feature/use_container_spack-stack-1.9.2

* 'feature/use_container_spack-stack-1.9.2' of github.com:NOAA-EPIC/global-workflow-cloud:
  reverse few changes
  re-sync with EMC repo
  Add master grib files to GFS HPSS archive for AIGFS (NOAA-EMC#4203)
  Update Snow filenames to comply with EE2 (NOAA-EMC#4195)
  Rename files for JEDI atm EE2 (NOAA-EMC#4193)
  Generate `pres_b` files for `RUN=gdas` and update `APCP` to `598`. (NOAA-EMC#4196)
  Update checks for MOM6 restarts when performing a re-run on failure (NOAA-EMC#4179)
  Decrease HPSS storage for GFS retros and address hpss bugs (NOAA-EMC#4184)
  Add noaacloud to ufsda case in dev/ush/load_modules (NOAA-EMC#4198)
  Remove replay from global workflow  (NOAA-EMC#4182)
  Add IODA stats text file to COM (NOAA-EMC#4176)
  Update UFS_UTILS submodule (NOAA-EMC#4178)
  Atm COM File Rename to Standardized Form  (NOAA-EMC#4117)
  Replace cp with cpfs/cpreq for atomic copies to COM directories (NOAA-EMC#4130)
  Create a UPP module for the global workflow (NOAA-EMC#4174)
  Refactor marine DA tasks (NOAA-EMC#4160)
  Delay ocean post-processing trigger to next-next forecast (NOAA-EMC#4167)
  Make options hashes
  Remove multiple option from static data template
  Fix static_data yaml (descriptions and labels)
  Fix static_data yaml (remove colon)
  Add Ursa to and remove C5 from list of HPCs in the bug report template (NOAA-EMC#4164)
  Rename marine (ocean/ice) files following EE2 conventions (NOAA-EMC#4162)
  Add attributes to Gaussian grid sfcanl file (NOAA-EMC#4149)
  Remove the snow analysis from archive (NOAA-EMC#4157)
  Update verif-global to fix pcp failures on special cases (NOAA-EMC#4154)
  Add CRTM fix directory paths to global-workflow (NOAA-EMC#4143)
  Update UFS Model  (NOAA-EMC#4138)
  Add functionality to assimilate the new snow observations (NOAA-EMC#4132)
DavidNew-NOAA added a commit to NOAA-EMC/GDASApp that referenced this pull request Jan 6, 2026
This PR, and companion to #1944
and NOAA-EMC/global-workflow#4160, does a few
things:

1. It adds templates required for rendering the JEDI marine LETKF
`local_ensemble_da` algorithm.
2. It adds a jinja switch to the marine obs YAMLs to turn on a halo
distribution when running marine LETKF.
3. It adds the `marine_` prefix to two or three JCB variables.
DavidNew-NOAA added a commit to NOAA-EMC/GDASApp that referenced this pull request Jan 16, 2026
# Description

This PR does a couple things:

1. It consolidates all parm YAML files required for marine DA tasks into
single master files for each task.
2. It adds parameters to the marine `jcb-base.yaml.j2` required for the
marine LETKF templates to be rendered by JCB>
3. It removes the `MARINE_` prefix from `MARINE_WINDOW_*` input
environment variables since these variables are now defined more
generally in the new Python `Analysis` class in GW.

# Companion PRs

NOAA-EMC/jcb-gdas#194
NOAA-EMC/global-workflow#4160

# Issues

...

# Automated CI tests to run in Global Workflow

Will be tested with full GW CI suite with results posted to GW companion
PR
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI-Gaeac6-Passed **Bot use only** CI testing on Gaea C6 for this PR has completed successfully CI-Ursa-Passed **Bot use only** CI testing on Ursa for this PR has completed successfully

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refactoring marine DA tasks

8 participants