Update AWS defaults for running obs prep jobs on cloud#3681
Merged
aerorahul merged 10 commits intoMay 14, 2025
Conversation
update to develop
TaylorRoper-NOAA
requested review from
DavidHuber-NOAA,
KateFriedman-NOAA,
WalterKolczynski-NOAA and
aerorahul
as code owners
2 tasks
TaylorRoper-NOAA
changed the title
KateFriedman-NOAA
approved these changes
May 13, 2025
Contributor
KateFriedman-NOAA
left a comment
KateFriedman-NOAA
left a comment
There was a problem hiding this comment.
Looks good, thanks @TaylorRoper-NOAA !
13 tasks
aerorahul
pushed a commit
that referenced
this pull request
May 14, 2025
This PR adds Fit2Obs to `modulefiles/module_base.noaacloud.lua` . Fit2Obs was updated to support NOAA cloud via NOAA-EMC/Fit2Obs#35. Fit2Obs is installed in `/contrib` filesystem at `/contrib/git` on AWS for QOSAP and EPIC. This is consistent with the choice for `BASE_GIT` made in #3681. Resolves #3682
tsga
added a commit
to tsga/global-workflow
that referenced
this pull request
May 15, 2025
* develop: Move parm/config/sfs/config.globus to dev/parm/config/sfs (NOAA-EMC#3697) Add Fit2Obs to modulefiles/module_base.noaacloud.lua (NOAA-EMC#3695) Update AWS defaults for running obs prep jobs on cloud (NOAA-EMC#3681) Add GCAFS forecast-only mode to the workflow Adds marine DA ensstat files to archiving (NOAA-EMC#3631) Migrate to mercury for globus/hpss transfers from MSU (NOAA-EMC#3655) STY: Remove empty __init.py__ files. (NOAA-EMC#3691) Relocate config templates to `dev/` space in prep for EE2 (NOAA-EMC#3684)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR enables running prep jobs on AWS. This required supporting
obsprocandprepobson the cloud (NOAA-EMC/obsproc#99 and NOAA-EMC/prepobs#43) and deciding on placement ofBASE_GITdirectory which housesobsprocandprepobs.BASE_GITis proposed to be/contrib/giton AWS (and other NOAA CSPs), andobsprocandprepobshave been cloned and installed in/contrib/giton the EPIC's AWS/contribfilesystem for future CI testing. The obs data used was made available on AWS via #3494, and theDMPDIRis defaulted to/bucket/dumpor user-defined viaDATA_MOUNTfor AWS. Since restricted data is not yet supported on NOAA CSPs, AWS defaults are updated to includeCHGRP_RSTPROD='NO'andMAKE_NSSTBUFR='YES'. UpdatingCHGRP_RSTPRODis necessary since NSST buffer files are assumed inobsprocto consist of restricted data and are overwritten as empty whenchgrp rstprodfails. Allowing making the NSST buffer file on the fly withCHGRP_RSTPROD='NO'was done via NOAA-EMC/obsproc#100.Resolves #3497
Type of change
Change characteristics
How has this been tested?
Tested on AWS and Hera by running the
C96_atm3DVarCI's starting half cycle andgfs_prepandgdas_prepfor first full cycle. On Hera, configuration was updated toMAKE_NSSTBUFR='YES'for comparison. Resulting logs,prepbufrfiles, andnsstbufrfiles can be found on Hera at/scratch1/BMC/qosap/Taylor.Roper/shared/prep_tests. The end of the file names indicate whether they are from the cloud or Hera.Checklist