Skip to content

Update AWS defaults for running obs prep jobs on cloud #3681

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aerorahul merged 10 commits into from
May 14, 2025
Merged

Update AWS defaults for running obs prep jobs on cloud #3681

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
a94f332
add dump dir; enable loading prepobs and fit2obs
TaylorRoper-NOAA Apr 24, 2025
fd08e35
fix accidental edit
TaylorRoper-NOAA Apr 28, 2025
5e57a0d
Merge branch 'develop' into feature/prep_jobs_aws
TaylorRoper-NOAA Apr 28, 2025
b1b06cc
enabling MAKE_NSSTBUFR on cloud
TaylorRoper-NOAA May 1, 2025
9c9d9b1
Merge branch 'develop' into feature/prep_jobs_aws
TaylorRoper-NOAA May 6, 2025
9254e2c
update with develop
TaylorRoper-NOAA May 9, 2025
d82b28c
updating awspw host yaml file for prep job configurations on cloud
TaylorRoper-NOAA May 12, 2025
5c5dca4
removing loading fit2obs on noaalcoud; update in separate PR
TaylorRoper-NOAA May 12, 2025
e8bdcd7
Merge branch 'develop' into feature/prep_jobs_aws
aerorahul May 14, 2025
319efdb
Merge branch 'develop' into feature/prep_jobs_aws
aerorahul May 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 4 additions & 6 deletions dev/workflow/hosts/awspw.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Paths
DMPDIR: '' # TODO: This does not yet exist.
BASE_GIT: '' #TODO: This does not yet exist.
DMPDIR: '${DATA_MOUNT:-/bucket}/dump'
BASE_GIT: '/contrib/git'
BASE_DATA: '/bucket/global-workflow-shared-data'
BASE_IC: '/bucket/global-workflow-shared-data/ICSDIR'
AERO_INPUTS_DIR: /contrib/global-workflow-shared-data/data/GEFS_ExtData/20250310
Expand All @@ -15,19 +15,17 @@ SCHEDULER: slurm
QUEUE: batch
PARTITION_BATCH: compute
PARTITION_SERVICE: process
CHGRP_RSTPROD: 'YES'
CHGRP_CMD: 'chgrp rstprod' # TODO: This is not yet supported.
# HPSS properties
HPSS_PROJECT: emc-global #TODO: See `ATARDIR` below.
ARCHCOM_TO: 'local'
ATARDIR: '' # TODO: This will not yet work from AWS.
CHGRP_RSTPROD: 'YES'
CHGRP_RSTPROD: 'NO' #TODO: rstprod group not yet supported.
CHGRP_CMD: 'chgrp rstprod' # TODO: This is not yet supported.
# Features
SUPPORTED_RESOLUTIONS: ['C48', 'C96', 'C192', 'C384', 'C768'] # TODO: Test and support all cubed-sphere resolutions.
DO_ARCHCOM: 'NO'
DO_TRACKER: 'NO'
DO_GENESIS: 'NO'
DO_METP: 'NO'
MAKE_NSSTBUFR: 'NO'
MAKE_NSSTBUFR: 'YES'
MAKE_ACFTBUFR: 'NO'
5 changes: 2 additions & 3 deletions modulefiles/module_base.noaacloud.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,8 @@ load(pathJoin("py-xarray", (os.getenv("py_xarray_ver") or "None")))
setenv("WGRIB2","wgrib2")
setenv("UTILROOT",(os.getenv("prod_util_ROOT") or "None"))

--prepend_path("MODULEPATH", pathJoin("/scratch1/NCEPDEV/global/glopara/git/prepobs/v" .. (os.getenv("prepobs_run_ver") or "None"), "modulefiles"))
--prepend_path("MODULEPATH", pathJoin("/scratch1/NCEPDEV/global/glopara/git/prepobs/feature-GFSv17_com_reorg_log_update/modulefiles"))
--load(pathJoin("prepobs", (os.getenv("prepobs_run_ver") or "None")))
prepend_path("MODULEPATH", "/contrib/git/prepobs/modulefiles")
load(pathJoin("prepobs", (os.getenv("prepobs_run_ver") or "None")))

--prepend_path("MODULEPATH", pathJoin("/scratch1/NCEPDEV/global/glopara/git/Fit2Obs/v" .. (os.getenv("fit2obs_ver") or "None"), "modulefiles"))
--load(pathJoin("fit2obs", (os.getenv("fit2obs_ver") or "None")))
Expand Down