Skip to content

Cleanup job for GEFS#2919

Merged
aerorahul merged 16 commits into
NOAA-EMC:developfrom
AntonMFernando-NOAA:cleanup/gefs
Sep 14, 2024
Merged

Cleanup job for GEFS#2919
aerorahul merged 16 commits into
NOAA-EMC:developfrom
AntonMFernando-NOAA:cleanup/gefs

Conversation

@AntonMFernando-NOAA
Copy link
Copy Markdown
Contributor

@AntonMFernando-NOAA AntonMFernando-NOAA commented Sep 13, 2024

Description

As forecast ensemble jobs are added to the global workflow, this PR ensures the output is being cleaned up properly once it is no longer needed.
Resolves #833

Type of change

  • A cleanup job that properly removes old ensemble data once it is no longer needed.
  • Add the cleanup job to the rocoto mesh.

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO

How has this been tested?

  • CI test on HERCULES

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • I have made corresponding changes to the documentation if necessary

@AntonMFernando-NOAA AntonMFernando-NOAA marked this pull request as ready for review September 13, 2024 19:27
Copy link
Copy Markdown
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good.
just a couple of comments.

Comment thread parm/config/gefs/config.cleanup Outdated
Comment thread parm/config/gefs/config.cleanup Outdated
Copy link
Copy Markdown
Contributor

@WalterKolczynski-NOAA WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will run this through the normal CI tests when it is complete, but to test it you'll need to run several cycles to test the cleanup bit properly. We don't keep a bunch of GEFS initial conditions around, so work with the ensemble team to set that up.

Comment thread workflow/rocoto/gefs_tasks.py Outdated
Copy link
Copy Markdown
Contributor

@WalterKolczynski-NOAA WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conditionally approved pending CI and manual completion of a longer GEFS test.

@aerorahul aerorahul merged commit b7f71e0 into NOAA-EMC:develop Sep 14, 2024
DavidHuber-NOAA added a commit to DavidHuber-NOAA/global-workflow that referenced this pull request Sep 16, 2024
* origin/develop:
  Update config.resources for bufr sounding job postsnd (NOAA-EMC#2917)
  Cleanup job for GEFS (NOAA-EMC#2919)
  Build GDASApp and unset memory in Gaea-C5 xml files (NOAA-EMC#2912)
  add 1 deg ocean/ice info to parm/config/gfs/config.resources (NOAA-EMC#2922)
  Support gefs C48 on Azure (NOAA-EMC#2881)
  Disable native grid writes for non-JEDI experiments; update C384 compression options (NOAA-EMC#2914)
DavidHuber-NOAA added a commit to DavidHuber-NOAA/global-workflow that referenced this pull request Sep 16, 2024
* emc/develop:
  Update config.resources for bufr sounding job postsnd (NOAA-EMC#2917)
  Cleanup job for GEFS (NOAA-EMC#2919)
  Build GDASApp and unset memory in Gaea-C5 xml files (NOAA-EMC#2912)
  add 1 deg ocean/ice info to parm/config/gfs/config.resources (NOAA-EMC#2922)
  Support gefs C48 on Azure (NOAA-EMC#2881)
  Disable native grid writes for non-JEDI experiments; update C384 compression options (NOAA-EMC#2914)
  Add new UPP links to .gitignore (NOAA-EMC#2904)
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this pull request Sep 17, 2024
…al_obs_path

* origin/develop:
  Update global atmos upp job to use COMIN/COMOUT (NOAA-EMC#2867)
  Update config.resources for bufr sounding job postsnd (NOAA-EMC#2917)
  Cleanup job for GEFS (NOAA-EMC#2919)
  Build GDASApp and unset memory in Gaea-C5 xml files (NOAA-EMC#2912)
  add 1 deg ocean/ice info to parm/config/gfs/config.resources (NOAA-EMC#2922)
  Support gefs C48 on Azure (NOAA-EMC#2881)
  Disable native grid writes for non-JEDI experiments; update C384 compression options (NOAA-EMC#2914)
bbakernoaa pushed a commit to bbakernoaa/global-workflow that referenced this pull request Sep 19, 2024
As forecast ensemble jobs are added to the global workflow, this PR
ensures the output is being cleaned up properly once it is no longer
needed.
  Resolves NOAA-EMC#833
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update cleanup job to scrub ensemble data

3 participants