Skip to content

Standardize static data across Tier-1 platforms; fix and improve IC and LBC data retrieval #744

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mkavulich merged 36 commits into from
May 3, 2022
Merged

Standardize static data across Tier-1 platforms; fix and improve IC and LBC data retrieval #744

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
c499652
Start reorganization of static data per https://github.com/ufs-commun…
mkavulich Apr 5, 2022
bd6e44d
NOMADS wget commands now will only retry once after failure to speed …
mkavulich Apr 5, 2022
d33ad19
Address 2 main issues for cycling failures.
christinaholtNOAA Apr 19, 2022
1a08664
Source the machine file before config.sh
christinaholtNOAA Apr 19, 2022
2b99615
Adding data stores to machine files.
christinaholtNOAA Apr 19, 2022
8194cff
Make data stores configurable.
christinaholtNOAA Apr 19, 2022
4dfdd87
Ensure that the tests use only data on disk.
christinaholtNOAA Apr 19, 2022
74e3726
Revert "Source the machine file before config.sh"
mkavulich Apr 19, 2022
8f77203
Christina updates
mkavulich Apr 20, 2022
1d5b021
Remove requirement for sfc files with LBCS.
christinaholtNOAA Apr 20, 2022
3dcdde3
Fix how templates of paths are handled.
christinaholtNOAA Apr 20, 2022
6622ded
Explicitly set name of RAP/HRRR files to match disk.
christinaholtNOAA Apr 20, 2022
2b295d5
Set filenames for RAP/HRRR
christinaholtNOAA Apr 20, 2022
d6e5a80
Fix typo
mkavulich Apr 20, 2022
be8e189
Specify location of pregen data for Cheyenne
mkavulich Apr 21, 2022
6fd909e
Staged static directory
Apr 22, 2022
f6c23ee
Standarized static files following PR #744 for Odin and Stampede
ywangwof Apr 23, 2022
04c53c2
Missed one update for Jet
mkavulich Apr 25, 2022
b2d18df
Add config override for data stores on all platforms.
christinaholtNOAA Apr 26, 2022
b3b02ee
updates for gaea
mark-a-potts Apr 27, 2022
3fd79b0
Remove capture_output for WCOSS compatibility.
christinaholtNOAA Apr 27, 2022
1715e27
Merge branch 'feature/standardize_static_data' of https://github.com/…
mark-a-potts Apr 27, 2022
2b69690
Update COMIN location on Hera to match other platforms
mkavulich Apr 27, 2022
feacafe
Fix Issue #700.
christinaholtNOAA Apr 20, 2022
beb1c1c
Updated noaacloud.sh
mark-a-potts Apr 27, 2022
5265c26
Merge branch 'feature/standardize_static_data' of https://github.com/…
mark-a-potts Apr 27, 2022
0d17d8b
Updates to standardize data locations on wcoss_dell_p3
mkavulich Apr 28, 2022
c5bebe7
A few wcoss changes requested by Chan-Hoo
mkavulich Apr 28, 2022
513cf20
getting conda working correctly
mark-a-potts Apr 28, 2022
e5bf146
Merge branch 'feature/standardize_static_data' of https://github.com/…
mark-a-potts Apr 28, 2022
9a3315e
replaced srun with mpiexec due to changes in image
mark-a-potts Apr 28, 2022
0279902
Merge remote-tracking branch 'upstream/develop' into feature/standard…
mkavulich May 3, 2022
cec308a
Correct location of Orion static data
mkavulich May 3, 2022
b4a7a18
Move DOMAIN_PREGEN_BASEDIR out of NCO section since it is no longer N…
mkavulich May 3, 2022
59f0d60
Fix default location for input model files on Cheyenne
mkavulich May 3, 2022
854d3f2
Revert "Fix default location for input model files on Cheyenne"
mkavulich May 3, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions scripts/exregional_get_extrn_mdl_files.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ elif [ "${ICS_OR_LBCS}" = "LBCS" ]; then
input_file_path=${EXTRN_MDL_SOURCE_BASEDIR_LBCS:-$EXTRN_MDL_SYSBASEDIR_LBCS}
fi

data_stores="hpss aws"
data_stores="${EXTRN_MDL_DATA_STORES}"

yyyymmddhh=${extrn_mdl_cdate:0:10}
yyyy=${yyyymmddhh:0:4}
Expand Down Expand Up @@ -141,7 +141,7 @@ if [ -n "${file_names:-}" ] ; then
fi

if [ -n "${input_file_path:-}" ] ; then
data_stores="disk hpss aws"
data_stores="disk $data_stores"
additional_flags="$additional_flags \
--input_file_path ${input_file_path}"
fi
Expand Down
22 changes: 11 additions & 11 deletions tests/WE2E/run_WE2E_tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -995,6 +995,9 @@ PTMP=\"${PTMP}\""
#
if [ "${USE_USER_STAGED_EXTRN_FILES}" = "TRUE" ]; then

# Ensure we only check on disk for these files
data_stores="disk"

extrn_mdl_source_basedir=${TEST_EXTRN_MDL_SOURCE_BASEDIR:-}
if [ ! -d "${extrn_mdl_source_basedir:-}" ] ; then
print_err_msg_exit "\
Expand All @@ -1005,20 +1008,16 @@ machine (MACHINE):
fi
EXTRN_MDL_SOURCE_BASEDIR_ICS="${extrn_mdl_source_basedir}/${EXTRN_MDL_NAME_ICS}"
if [ "${EXTRN_MDL_NAME_ICS}" = "FV3GFS" ] ; then
EXTRN_MDL_SOURCE_BASEDIR_ICS="${EXTRN_MDL_SOURCE_BASEDIR_ICS}/${FV3GFS_FILE_FMT_ICS}/\${DATE_FIRST_CYCL}\${CYCL_HRS[0]}"
elif [ "${EXTRN_MDL_NAME_ICS}" = "NAM" ] ; then
EXTRN_MDL_SOURCE_BASEDIR_ICS="${EXTRN_MDL_SOURCE_BASEDIR_ICS}/\${DATE_FIRST_CYCL}\${CYCL_HRS[0]}/for_ICS"
EXTRN_MDL_SOURCE_BASEDIR_ICS="${EXTRN_MDL_SOURCE_BASEDIR_ICS}/${FV3GFS_FILE_FMT_ICS}/\${yyyymmddhh}"
Comment thread
gsketefian marked this conversation as resolved.
else
EXTRN_MDL_SOURCE_BASEDIR_ICS="${EXTRN_MDL_SOURCE_BASEDIR_ICS}/\${DATE_FIRST_CYCL}\${CYCL_HRS[0]}"
EXTRN_MDL_SOURCE_BASEDIR_ICS="${EXTRN_MDL_SOURCE_BASEDIR_ICS}/\${yyyymmddhh}"
fi

EXTRN_MDL_SOURCE_BASEDIR_LBCS="${extrn_mdl_source_basedir}/${EXTRN_MDL_NAME_LBCS}"
if [ "${EXTRN_MDL_NAME_LBCS}" = "FV3GFS" ] ; then
EXTRN_MDL_SOURCE_BASEDIR_LBCS="${EXTRN_MDL_SOURCE_BASEDIR_LBCS}/${FV3GFS_FILE_FMT_LBCS}/\${DATE_FIRST_CYCL}\${CYCL_HRS[0]}"
elif [ "${EXTRN_MDL_NAME_LBCS}" = "GSMGFS" ] ; then
EXTRN_MDL_SOURCE_BASEDIR_LBCS="${EXTRN_MDL_SOURCE_BASEDIR_LBCS}/\${DATE_FIRST_CYCL}\${CYCL_HRS[0]}"
EXTRN_MDL_SOURCE_BASEDIR_LBCS="${EXTRN_MDL_SOURCE_BASEDIR_LBCS}/${FV3GFS_FILE_FMT_LBCS}/\${yyyymmddhh}"
else
EXTRN_MDL_SOURCE_BASEDIR_LBCS="${EXTRN_MDL_SOURCE_BASEDIR_LBCS}/\${DATE_FIRST_CYCL}\${CYCL_HRS[0]}/for_LBCS"
EXTRN_MDL_SOURCE_BASEDIR_LBCS="${EXTRN_MDL_SOURCE_BASEDIR_LBCS}/\${yyyymmddhh}"
fi
#
# Make sure that the forecast length is evenly divisible by the interval
Expand All @@ -1039,10 +1038,11 @@ boundary conditions specification interval (LBC_SPEC_INTVL_HRS):
# Locations and names of user-staged external model files for generating
# ICs and LBCs.
#
EXTRN_MDL_SOURCE_BASEDIR_ICS=${EXTRN_MDL_SOURCE_BASEDIR_ICS}
EXTRN_MDL_SOURCE_BASEDIR_ICS='${EXTRN_MDL_SOURCE_BASEDIR_ICS}'
EXTRN_MDL_FILES_ICS=( ${EXTRN_MDL_FILES_ICS[@]} )
EXTRN_MDL_SOURCE_BASEDIR_LBCS=${EXTRN_MDL_SOURCE_BASEDIR_LBCS}
EXTRN_MDL_FILES_LBCS=( ${EXTRN_MDL_FILES_LBCS[@]} )"
EXTRN_MDL_SOURCE_BASEDIR_LBCS='${EXTRN_MDL_SOURCE_BASEDIR_LBCS}'
EXTRN_MDL_FILES_LBCS=( ${EXTRN_MDL_FILES_LBCS[@]} )
EXTRN_MDL_DATA_STORES=\"$data_stores\""

fi
#
Expand Down
1 change: 1 addition & 0 deletions ...extrn_mdls_suites_community/config.grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_HRRR.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ EXTRN_MDL_NAME_ICS="FV3GFS"
FV3GFS_FILE_FMT_ICS="grib2"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200810"
DATE_LAST_CYCL="20200810"
Expand Down
2 changes: 2 additions & 0 deletions ..._mdls_suites_community/config.grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ CCPP_PHYS_SUITE="FV3_HRRR"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
Comment thread
gsketefian marked this conversation as resolved.
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200810"
DATE_LAST_CYCL="20200810"
Expand Down
2 changes: 2 additions & 0 deletions ...ites_community/config.grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ CCPP_PHYS_SUITE="FV3_RRFS_v1alpha"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200801"
DATE_LAST_CYCL="20200801"
Expand Down
2 changes: 2 additions & 0 deletions ...uites_community/config.grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ CCPP_PHYS_SUITE="FV3_RRFS_v1beta"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200801"
DATE_LAST_CYCL="20200801"
Expand Down
3 changes: 3 additions & 0 deletions ...mdls_suites_community/config.grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,13 @@ CCPP_PHYS_SUITE="FV3_HRRR"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="HRRR"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200810"
DATE_LAST_CYCL="20200810"
CYCL_HRS=( "00" )

FCST_LEN_HRS="24"
LBC_SPEC_INTVL_HRS="3"

2 changes: 2 additions & 0 deletions ...ites_community/config.grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_RRFS_v1beta.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ CCPP_PHYS_SUITE="FV3_RRFS_v1beta"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="HRRR"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200810"
DATE_LAST_CYCL="20200810"
Expand Down
2 changes: 2 additions & 0 deletions ..._mdls_suites_community/config.grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ CCPP_PHYS_SUITE="FV3_HRRR"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200810"
DATE_LAST_CYCL="20200810"
Expand Down
2 changes: 2 additions & 0 deletions ...ites_community/config.grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ CCPP_PHYS_SUITE="FV3_RRFS_v1alpha"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200810"
DATE_LAST_CYCL="20200810"
Expand Down
2 changes: 2 additions & 0 deletions ...uites_community/config.grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ CCPP_PHYS_SUITE="FV3_RRFS_v1beta"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200810"
DATE_LAST_CYCL="20200810"
Expand Down
2 changes: 2 additions & 0 deletions ...s_suites_community/config.grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_GFS_v15p2.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ CCPP_PHYS_SUITE="FV3_GFS_v15p2"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200801"
DATE_LAST_CYCL="20200801"
Expand Down
3 changes: 3 additions & 0 deletions ...n_mdls_suites_community/config.grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@ CCPP_PHYS_SUITE="FV3_HRRR"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )


DATE_FIRST_CYCL="20200810"
DATE_LAST_CYCL="20200810"
Expand Down
2 changes: 2 additions & 0 deletions ...uites_community/config.grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ CCPP_PHYS_SUITE="FV3_RRFS_v1alpha"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200801"
DATE_LAST_CYCL="20200801"
Expand Down
2 changes: 2 additions & 0 deletions ...suites_community/config.grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ CCPP_PHYS_SUITE="FV3_RRFS_v1beta"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200801"
DATE_LAST_CYCL="20200801"
Expand Down
2 changes: 2 additions & 0 deletions ..._mdls_suites_community/config.grid_RRFS_SUBCONUS_3km_ics_HRRR_lbcs_RAP_suite_GFS_v15p2.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ CCPP_PHYS_SUITE="FV3_GFS_v15p2"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200801"
DATE_LAST_CYCL="20200801"
Expand Down
2 changes: 2 additions & 0 deletions ...rn_mdls_suites_nco/config.nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ CCPP_PHYS_SUITE="FV3_HRRR"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200810"
DATE_LAST_CYCL="20200810"
Expand Down
2 changes: 2 additions & 0 deletions tests/WE2E/test_configs/wflow_features/config.get_from_HPSS_ics_HRRR_lbcs_RAP.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ CCPP_PHYS_SUITE="FV3_HRRR"

EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200801"
DATE_LAST_CYCL="20200801"
Expand Down
2 changes: 2 additions & 0 deletions tests/WE2E/test_configs/wflow_features/config.specify_DT_ATMOS_LAYOUT_XY_BLOCKSIZE.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ CCPP_PHYS_SUITE="FV3_HRRR"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200801"
DATE_LAST_CYCL="20200801"
Expand Down
2 changes: 2 additions & 0 deletions tests/WE2E/test_configs/wflow_features/config.subhourly_post.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ CCPP_PHYS_SUITE="FV3_RRFS_v1beta"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200810"
DATE_LAST_CYCL="20200810"
Expand Down
2 changes: 2 additions & 0 deletions tests/WE2E/test_configs/wflow_features/config.subhourly_post_ensemble_2mems.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ CCPP_PHYS_SUITE="FV3_RRFS_v1beta"
EXTRN_MDL_NAME_ICS="HRRR"
EXTRN_MDL_NAME_LBCS="RAP"
USE_USER_STAGED_EXTRN_FILES="TRUE"
EXTRN_MDL_FILES_ICS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )
EXTRN_MDL_FILES_LBCS=( '{yy}{jjj}{hh}00{fcst_hr:02d}00' )

DATE_FIRST_CYCL="20200810"
DATE_LAST_CYCL="20200810"
Expand Down
12 changes: 6 additions & 6 deletions ush/NOMADS_get_extrn_mdl_files.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,10 @@ cd gfs.$yyyymmdd/$hh

#getting online analysis data
if [ $file_fmt == "grib2" ] || [ $file_fmt == "GRIB2" ]; then
wget -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.pgrb2.0p25.f000
wget --tries=2 -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.pgrb2.0p25.f000
else
wget -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.atmanl.nemsio
wget -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.sfcanl.nemsio
wget --tries=2 -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.atmanl.nemsio
wget --tries=2 -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.sfcanl.nemsio
fi

#getting online forecast data
Expand All @@ -60,10 +60,10 @@ echo $ifcst
echo $ifcst_str
#
if [ $file_fmt == "grib2" ] || [ $file_fmt == "GRIB2" ]; then
wget -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.pgrb2.0p25.f${ifcst_str}
wget --tries=2 -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.pgrb2.0p25.f${ifcst_str}
else
wget -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.atmf${ifcst_str}.nemsio
wget -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.sfcf${ifcst_str}.nemsio
wget --tries=2 -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.atmf${ifcst_str}.nemsio
wget --tries=2 -c https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.$yyyymmdd/$hh/gfs.t${hh}z.sfcf${ifcst_str}.nemsio
fi
#
ifcst=$[$ifcst+$nfcst_int]
Expand Down
41 changes: 27 additions & 14 deletions ush/config_defaults.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -238,19 +238,6 @@ EXEC_SUBDIR="bin"
# need in order to create initial and boundary condition files for a given
# cycle on the native FV3-LAM grid.
#
# FIXLAM_NCO_BASEDIR:
# The base directory containing pregenerated grid, orography, and surface
# climatology files. For the pregenerated grid specified by PREDEF_GRID_NAME,
# these "fixed" files are located in:
#
# ${FIXLAM_NCO_BASEDIR}/${PREDEF_GRID_NAME}
#
# The workflow scripts will create a symlink in the experiment directory
# that will point to a subdirectory (having the name of the grid being
# used) under this directory. This variable should be set to a null
# string in this file, but it can be specified in the user-specified
# workflow configuration file (EXPT_CONFIG_FN).
#
# envir, NET, model_ver, RUN:
# Standard environment variables defined in the NCEP Central Operations WCOSS
# Implementation Standards document as follows:
Expand Down Expand Up @@ -292,7 +279,7 @@ EXEC_SUBDIR="bin"
#-----------------------------------------------------------------------
#
COMIN="/path/of/directory/containing/data/files/for/IC/LBCS"
FIXLAM_NCO_BASEDIR=""
STMP="/base/path/of/directory/containing/model/input/and/raw/output/files"
envir="para"
NET="rrfs"
model_ver="v1.0.0"
Expand Down Expand Up @@ -711,13 +698,21 @@ EXTRN_MDL_SYSBASEDIR_LBCS=''
# EXTRN_MDL_FILES_LBCS:
# Analogous to EXTRN_MDL_FILES_ICS but for LBCs instead of ICs.
#
# EXTRN_MDL_DATA_STORES:
# A list of data stores where the scripts should look for external model
# data. The list is in priority order. If disk information is provided
# via USE_USER_STAGED_EXTRN_FILES or a known location on the platform,
# the disk location will be highest priority. Options are disk, hpss,
# aws, and nomads.
#
#-----------------------------------------------------------------------
#
USE_USER_STAGED_EXTRN_FILES="FALSE"
EXTRN_MDL_SOURCE_BASEDIR_ICS=""
EXTRN_MDL_FILES_ICS=""
EXTRN_MDL_SOURCE_BASEDIR_LBCS=""
EXTRN_MDL_FILES_LBCS=""
EXTRN_MDL_DATA_STORES=""
#
#-----------------------------------------------------------------------
#
Expand Down Expand Up @@ -1305,6 +1300,22 @@ VX_ENSPOINT_PROB_TN="run_enspointvx_prob"
# SFC_CLIMO_DIR:
# Same as GRID_DIR but for the MAKE_SFC_CLIMO_TN task.
#
# DOMAIN_PREGEN_BASEDIR:
# The base directory containing pregenerated grid, orography, and surface
# climatology files. This is an alternative for setting GRID_DIR,
# OROG_DIR, and SFC_CLIMO_DIR individually
#
# For the pregenerated grid specified by PREDEF_GRID_NAME,
# these "fixed" files are located in:
#
# ${DOMAIN_PREGEN_BASEDIR}/${PREDEF_GRID_NAME}
#
# The workflow scripts will create a symlink in the experiment directory
# that will point to a subdirectory (having the name of the grid being
# used) under this directory. This variable should be set to a null
# string in this file, but it can be specified in the user-specified
# workflow configuration file (EXPT_CONFIG_FN).
#
# RUN_TASK_GET_EXTRN_ICS:
# Flag that determines whether the GET_EXTRN_ICS_TN task is to be run.
#
Expand Down Expand Up @@ -1351,6 +1362,8 @@ OROG_DIR="/path/to/pregenerated/orog/files"
RUN_TASK_MAKE_SFC_CLIMO="TRUE"
SFC_CLIMO_DIR="/path/to/pregenerated/surface/climo/files"

DOMAIN_PREGEN_BASEDIR=""

RUN_TASK_GET_EXTRN_ICS="TRUE"
RUN_TASK_GET_EXTRN_LBCS="TRUE"
RUN_TASK_MAKE_ICS="TRUE"
Expand Down
25 changes: 13 additions & 12 deletions ush/machine/cheyenne.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,12 +46,13 @@ QUEUE_HPSS=${QUEUE_HPSS:-"regular"}
QUEUE_FCST=${QUEUE_FCST:-"regular"}

# UFS SRW App specific paths
FIXgsm=${FIXgsm:-"/glade/p/ral/jntp/UFS_CAM/fix/fix_am"}
FIXaer=${FIXaer:-"/glade/p/ral/jntp/UFS_CAM/fix/fix_aer"}
FIXlut=${FIXlut:-"/glade/p/ral/jntp/UFS_CAM/fix/fix_lut"}
TOPO_DIR=${TOPO_DIR:-"/glade/p/ral/jntp/UFS_CAM/fix/fix_orog"}
SFC_CLIMO_INPUT_DIR=${SFC_CLIMO_INPUT_DIR:-"/glade/p/ral/jntp/UFS_CAM/fix/climo_fields_netcdf"}
FIXLAM_NCO_BASEDIR=${FIXLAM_NCO_BASEDIR:-"/needs/to/be/specified"}
staged_data_dir="/glade/p/ral/jntp/UFS_SRW_App/develop"
Comment thread
gsketefian marked this conversation as resolved.
FIXgsm=${FIXgsm:-"${staged_data_dir}/fix/fix_am"}
FIXaer=${FIXaer:-"${staged_data_dir}/fix/fix_aer"}
FIXlut=${FIXlut:-"${staged_data_dir}/fix/fix_lut"}
TOPO_DIR=${TOPO_DIR:-"${staged_data_dir}/fix/fix_orog"}
SFC_CLIMO_INPUT_DIR=${SFC_CLIMO_INPUT_DIR:-"${staged_data_dir}/fix/fix_sfc_climo"}
DOMAIN_PREGEN_BASEDIR=${DOMAIN_PREGEN_BASEDIR:-"${staged_data_dir}/FV3LAM_pregen"}

# Run commands for executables
RUN_CMD_SERIAL="time"
Expand All @@ -62,12 +63,12 @@ RUN_CMD_POST='mpirun -np $nprocs'
# MET/METplus-Related Paths
MET_INSTALL_DIR=${MET_INSTALL_DIR:-"/glade/p/ral/jntp/MET/MET_releases/10.0.0"}
METPLUS_PATH=${METPLUS_PATH:-"/glade/p/ral/jntp/MET/METplus/METplus-4.0.0"}
CCPA_OBS_DIR=${CCPA_OBS_DIR:-"/glade/p/ral/jntp/UFS_SRW_app/develop/obs_data/ccpa/proc"}
MRMS_OBS_DIR=${MRMS_OBS_DIR:-"/glade/p/ral/jntp/UFS_SRW_app/develop/obs_data/mrms/proc"}
NDAS_OBS_DIR=${NDAS_OBS_DIR:-"/glade/p/ral/jntp/UFS_SRW_app/develop/obs_data/ndas/proc"}
CCPA_OBS_DIR=${CCPA_OBS_DIR:-"${staged_data_dir}/obs_data/ccpa/proc"}
MRMS_OBS_DIR=${MRMS_OBS_DIR:-"${staged_data_dir}/obs_data/mrms/proc"}
NDAS_OBS_DIR=${NDAS_OBS_DIR:-"${staged_data_dir}/obs_data/ndas/proc"}
MET_BIN_EXEC=${MET_BIN_EXEC:-"bin"}

# Test Data Locations
TEST_PREGEN_BASEDIR="/glade/p/ral/jntp/UFS_SRW_app/FV3LAM_pregen"
TEST_COMIN="/glade/p/ral/jntp/UFS_SRW_app/COMGFS"
TEST_EXTRN_MDL_SOURCE_BASEDIR="/glade/p/ral/jntp/UFS_SRW_app/staged_extrn_mdl_files"
TEST_COMIN="${staged_data_dir}/COMGFS"
TEST_PREGEN_BASEDIR="${staged_data_dir}/FV3LAM_pregen"
TEST_EXTRN_MDL_SOURCE_BASEDIR="${staged_data_dir}/input_model_data"
Loading