Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
107 commits
Select commit Hold shift + click to select a range
90b7ada
Minor changes.
gsketefian Jul 28, 2021
f4b21d6
Bug fix to ensure that the is_element_of function works even when the…
gsketefian Jul 28, 2021
03e6bfb
Reorganize the way external model files are obtained. Details below.
gsketefian Jul 29, 2021
aee76a4
Fix typo in instructions for is_element_of; other minor changes to ba…
gsketefian Jul 30, 2021
13052da
Add utility function (parse_cdate) to parse a given cdate to obtain i…
gsketefian Jul 30, 2021
574df9d
Modify exregional_run_post.sh to use the new utility function parse_c…
gsketefian Jul 30, 2021
3ebc1b4
Bug fix.
gsketefian Jul 30, 2021
59b6289
Modify if-statement for setting user-staged directories and files so …
gsketefian Jul 30, 2021
e74a284
Modifications to get "user_dir" and "noaa_hpss" cases in EXTRN_MDL_DA…
gsketefian Jul 30, 2021
0965edd
Merge branch 'develop' into feature/reorg_get_extrn_files
gsketefian Jul 30, 2021
6a33e6a
Remove unneeded file.
gsketefian Jul 30, 2021
f0fd21f
Modifications to get "sys_dir" case in EXTRN_MDL_DATA_SOURCES array t…
gsketefian Aug 2, 2021
043a634
Minor changes.
gsketefian Aug 2, 2021
ed4dbfb
Fill in NetCDF sections.
gsketefian Aug 2, 2021
763e6be
Bug fixes and cleanup of unnecessary and commented-out code.
gsketefian Aug 2, 2021
a3ee815
Remove unnecessary execute permission on file.
gsketefian Aug 2, 2021
f5ecbde
Create new function (set_extrn_mdl_arcv_file_dir_names) that sets the…
gsketefian Aug 3, 2021
a16f574
Changes to get the workflow to fetch external model files from NOMADS…
gsketefian Aug 4, 2021
5a298a2
Remove unneeded argument.
gsketefian Aug 4, 2021
f30c9d0
In the functions that get external model data from various sources, u…
gsketefian Aug 4, 2021
637f0e5
Add category subdirectory containing WE2E tests for getting external …
gsketefian Aug 4, 2021
4c8740e
Add forgotten file.
gsketefian Aug 4, 2021
c6738ed
Add/remove stanzas for Cheyenne.
Aug 5, 2021
6a06314
Bug fix to FV3GFS source_subdir.
Aug 5, 2021
94eb6db
Bug fix to wget command.
Aug 5, 2021
6eca454
Fix indentation.
gsketefian Aug 5, 2021
69bd9f3
Clean up code and comments.
Aug 5, 2021
c619da8
Merge remote-tracking branch 'origin/feature/reorg_get_extrn_files' i…
gsketefian Aug 5, 2021
23b5494
Add in the error messages the calling script's or function's file nam…
gsketefian Aug 5, 2021
a725ee1
Shorten variable names, edit comments.
gsketefian Aug 5, 2021
9c9b8b9
Rename variables for clarity, clean up comments, some bug fixes.
gsketefian Aug 5, 2021
89bf750
For clarity, move all scripts in ush having to do with external model…
gsketefian Aug 6, 2021
6262ddf
Various changes described below:
gsketefian Aug 6, 2021
0958206
Rename WE2E test configuration files in the get_extrn_mdl_files categ…
gsketefian Aug 6, 2021
ca1f19c
Add approprate values of GET_EXTRN_MDL_DATA_SOURCES to WE2E configura…
gsketefian Aug 6, 2021
8616838
Move and rename a subset of the WE2E test configuration files in the …
gsketefian Aug 6, 2021
2bd4172
Move and rename more WE2E test configuration files in the "wflow_feat…
gsketefian Aug 6, 2021
40a6596
Update values of local variables; consolidate multiple repeat values …
gsketefian Aug 8, 2021
c56d5a9
Rename WE2E test configuration files to names that are easier to comp…
gsketefian Aug 12, 2021
f41c4c9
Add new WE2E files for getting external model files from disk in diff…
gsketefian Aug 12, 2021
03a0765
Combine "user_dir" and "sys_dir" cases in EXTRN_MDL_SOURCE_DATA into …
gsketefian Aug 12, 2021
cc18815
In WE2E tests category subdirectory "wflow_features", remove non-exis…
gsketefian Aug 17, 2021
e1fdba4
For clarity, rename some WE2E test configuration files.
gsketefian Aug 17, 2021
c0624a1
In the WE2E category subdirectory "get_extrn_mdl_files", in the test …
gsketefian Aug 17, 2021
0834e71
Update test in "get_extrn_mdl_files" category subdirectory; replace t…
gsketefian Aug 17, 2021
cdda9e2
Update to the test description.
gsketefian Aug 17, 2021
5eae15c
Various improvements. Details below.
gsketefian Aug 18, 2021
683380b
Bug fixes.
gsketefian Aug 18, 2021
80a66ce
Remove unneeded function arguments; add local variabl; clean up comme…
gsketefian Aug 18, 2021
4a0ab09
Bug fixes.
gsketefian Aug 18, 2021
1778163
Add limit on number of times to ping so that "ping" quits after that …
Aug 18, 2021
3b0659c
Add script to get files from NOMADS outside of the workflow but using…
Aug 18, 2021
c9eaa0f
Make the loop setups briefer (by using < instead of <=).
gsketefian Aug 18, 2021
e41ff78
Bug fix.
gsketefian Aug 18, 2021
70c747b
Edit error message.
gsketefian Aug 18, 2021
6ba7505
Generalize stand-alone script for getting external files from NOMADS …
gsketefian Aug 18, 2021
909e05c
Change the name of the wget log file so that there is a different log…
Aug 19, 2021
9d80bc0
Edit comments.
Aug 19, 2021
52e025b
Many improvement to the stand-alone script that gets FV3GFS grib2 fil…
Aug 19, 2021
65b02d9
More improvements of the stand-alone script for getting FV3GFS grib2 …
Aug 19, 2021
b8915e9
Even more improvements of the stand-alone script for getting FV3GFS g…
Aug 19, 2021
791cdd1
In check_for_preexist_dir_file.sh, add a default value of "FALSE" in …
Aug 20, 2021
36c5dbe
In print_input_args.sh, remove the use of the "-v" conditional operat…
Aug 20, 2021
65b8bff
In load_modules_run_task.sh, replace "source" with "." since that is …
Aug 20, 2021
a323cfb
Alphabetize the order of the sourced files.
Aug 20, 2021
c785aca
Add check to see if NOMADS is accessible; remove definition of VERBOS…
Aug 20, 2021
fc31926
Add new function that checks the current machine's acccess to NOMADS;…
Aug 20, 2021
d791cd7
Add optional argument "check_access" to the function get_extrn_mdl_fi…
Aug 20, 2021
c80a950
NOMADS-related changes (mostly). Details below.
gsketefian Aug 20, 2021
78244b9
For brevity, rename experiment variable LBC_SPEC_FCST_HRS to LBC_SPEC…
Aug 21, 2021
d6f3797
Add more informational messages.
Aug 21, 2021
cf01e9b
Add "none" to the list of possible values of preexisting_dir_method. …
Aug 23, 2021
38e38cf
Add new workflow variables RUN_TASK_GET_EXTRN_ICS and RUN_TASK_GET_EX…
Aug 23, 2021
6d1904d
Add two new arguments to the stand-alone script get_FV3GFS_grib2_file…
Aug 23, 2021
bd840fa
Fixes to comments.
Aug 23, 2021
1fac5f8
In valid_param_vals.sh, replace all the variables representing valid …
Aug 23, 2021
9bf807d
Turn off "set -x".
Aug 23, 2021
04d3cd6
Add set_boolean_to_TRUE_or_FALSE.sh to the list of utility functions …
Aug 23, 2021
a00f034
Make sure that boolean_value is set to a vaild value; bug fix -- b ne…
Aug 23, 2021
369dee4
Minor changes.
Aug 23, 2021
87bc5e1
Rearrange lines and bug fix of several variable names.
Aug 23, 2021
4d41671
Add new function create_extrn_mdl_var_defns_file.sh to create the var…
Aug 23, 2021
39693ca
Edits to comments and informational messages.
Aug 23, 2021
bbd1056
Change default values of EXTRN_MDL_BASEDIRS_[ICS|LBCS] and EXTRN_MDL_…
Aug 24, 2021
7739c92
Bug fix in argument name; edit comments.
Aug 24, 2021
664f5fc
Changes to get_FV3GFS_grib2_files_from_NOMADS.sh to make it work duri…
Aug 24, 2021
c44c37c
Call the stand-alone script for fetching FV3GFS grib2 files from NOMA…
Aug 24, 2021
03b90d5
Remove USE_USER_STAGED_EXTRN_MDL_FILES from WE2E test configuration f…
gsketefian Aug 24, 2021
45c280f
Turn off "set -x".
gsketefian Aug 24, 2021
2a189b1
Turn off "set -x" for brevity; fix indentation.
gsketefian Aug 24, 2021
1c384e2
Modify WE2E scripts according to match the rearrangement of the stage…
gsketefian Aug 24, 2021
9819eda
Set the default value of COMINgfs to an empty string because otherwis…
gsketefian Aug 24, 2021
021fc80
Bugfix in exregional_get_extrn_mdl_files.sh; improvement in run_WE2E_…
gsketefian Aug 25, 2021
e9ac6a3
Change variable names for clarity.
gsketefian Aug 25, 2021
67bae56
Rename WE2E test configuration file to be consistent with the names o…
gsketefian Aug 25, 2021
c582a1a
Bug fix.
gsketefian Aug 25, 2021
a41d25b
Remove script NOMADS_get_extrn_mdl_files.sh since it is now supersede…
gsketefian Aug 25, 2021
3f4d0a6
Merge branch 'develop' into feature/reorg_get_extrn_files
gsketefian Aug 25, 2021
920909d
Merge branch 'develop' into feature/reorg_get_extrn_files
gsketefian Aug 25, 2021
24d84d3
Bug fix -- PAZI needs to be defined (to some default value) for GFDLg…
gsketefian Aug 25, 2021
0c0f369
Rename WE2E tests for clarity.
gsketefian Aug 25, 2021
4446c2d
Use multiple days for the test that tries to fetch external model fil…
Aug 25, 2021
4d85311
Rename test to correctly represent what it does.
Aug 25, 2021
1371ee1
Modify setup.sh to require that EXTRN_MDL_DIR_FILE_LAYOUT be set to "…
gsketefian Aug 26, 2021
2522a2c
Change value of COMINgfs since apparently the old value no longer exi…
gsketefian Aug 26, 2021
7d72327
Add code to remove empty elements from EXTRN_MDL_BASEDIR_[ICS|LBCS], …
gsketefian Aug 26, 2021
dadde8e
Update external model directory on Jet.
gsketefian Oct 8, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 9 additions & 118 deletions jobs/JREGIONAL_GET_EXTRN_MDL_FILES
Original file line number Diff line number Diff line change
Expand Up @@ -33,15 +33,6 @@
#
#-----------------------------------------------------------------------
#
# Source the file defining the function that will be used to set various
# external-model-associated variables.
#
#-----------------------------------------------------------------------
#
. $USHDIR/get_extrn_mdl_file_dir_info.sh
#
#-----------------------------------------------------------------------
#
# Save current shell options (in a global array). Then set new options
# for this script/function.
#
Expand Down Expand Up @@ -167,128 +158,28 @@ check_var_valid_value "ICS_OR_LBCS" "valid_vals_ICS_OR_LBCS"
#
#-----------------------------------------------------------------------
#
# Set the parameter anl_or_fcst that determines whether we want to get
# analysis or forecast files. This depends on whether we want these files
# to generate initial condition and surface field files or lateral boundary
# condition files. Also, set time_offset_hrs, which is the offset in
# hours between the current cycle's starting time and the starting time
# of the external model providing the LBCs.
# Set the name of the directory in which to stage the external model
# files (or symlinks to such files) for the current cycle. Then create
# the directory (if it doesn't already exist) and change location to it.
#
#-----------------------------------------------------------------------
#
if [ "${ICS_OR_LBCS}" = "ICS" ]; then
anl_or_fcst="ANL"
time_offset_hrs="0"
elif [ "${ICS_OR_LBCS}" = "LBCS" ]; then
anl_or_fcst="FCST"
time_offset_hrs="${EXTRN_MDL_LBCS_OFFSET_HRS}"
fi
#
#-----------------------------------------------------------------------
#
# Set the name of and then create the directory in which to stage the
# external model files for the current cycle (if it doesn't already exist).
# Then change location to that directory.
#
#-----------------------------------------------------------------------
#
extrn_mdl_staging_dir="${CYCLE_DIR}/${EXTRN_MDL_NAME}/for_${ICS_OR_LBCS}"
mkdir_vrfy -p "${extrn_mdl_staging_dir}"
cd_vrfy "${extrn_mdl_staging_dir}"
#
#-----------------------------------------------------------------------
#
# Call the function that sets various external-model-associated variables.
# See the function defintion file for the definitions of these variables.
#
#-----------------------------------------------------------------------
#
get_extrn_mdl_file_dir_info \
extrn_mdl_name="${EXTRN_MDL_NAME}" \
anl_or_fcst="${anl_or_fcst}" \
cdate_FV3LAM="${CDATE}" \
time_offset_hrs="${time_offset_hrs}" \
varname_extrn_mdl_cdate="extrn_mdl_cdate" \
varname_extrn_mdl_lbc_spec_fhrs="extrn_mdl_lbc_spec_fhrs" \
varname_extrn_mdl_fns_on_disk="extrn_mdl_fns_on_disk" \
varname_extrn_mdl_fns_in_arcv="extrn_mdl_fns_in_arcv" \
varname_extrn_mdl_sysdir="extrn_mdl_sysdir" \
varname_extrn_mdl_arcv_fmt="extrn_mdl_arcv_fmt" \
varname_extrn_mdl_arcv_fns="extrn_mdl_arcv_fns" \
varname_extrn_mdl_arcv_fps="extrn_mdl_arcv_fps" \
varname_extrn_mdl_arcvrel_dir="extrn_mdl_arcvrel_dir" || \
print_err_msg_exit "\
Call to function get_extrn_mdl_file_dir_info failed."
#
#-----------------------------------------------------------------------
#
# Set the directory in which to check for the external model files (which
# we refer to here as the "source" directory) to the default one set above
# for the current machine and external model.
#
#-----------------------------------------------------------------------
#
extrn_mdl_source_dir="${extrn_mdl_sysdir}"
#
#-----------------------------------------------------------------------
#
# If the user has specified that the external model files to be used for
# generating ICs or LBCs are staged, then reset extrn_mdl_source_dir to
# the user-specified directory in which these files are staged, and reset
# extrn_mdl_fns_on_disk to the user-specified array containing the names
# of the files.
#
#-----------------------------------------------------------------------
#
if [ "${USE_USER_STAGED_EXTRN_FILES}" = "TRUE" ]; then

if [ "${ICS_OR_LBCS}" = "ICS" ]; then
extrn_mdl_source_dir="${EXTRN_MDL_SOURCE_BASEDIR_ICS}/$CDATE"
extrn_mdl_fns_on_disk=( $( printf "%s " "${EXTRN_MDL_FILES_ICS[@]}" ))
elif [ "${ICS_OR_LBCS}" = "LBCS" ]; then
extrn_mdl_source_dir="${EXTRN_MDL_SOURCE_BASEDIR_LBCS}/$CDATE"
extrn_mdl_fns_on_disk=( $( printf "%s " "${EXTRN_MDL_FILES_LBCS[@]}" ))
fi

if [ ! -d "${extrn_mdl_source_dir}" ]; then
print_err_msg_exit "\
The directory extrn_mdl_source_dir containing the user-staged external
model files does not exist:
extrn_mdl_source_dir = \"${extrn_mdl_source_dir}\"
Please ensure that the directory specified by extrn_mdl_source_dir exists
and that all the files specified in the array extrn_mdl_fns_on_disk exist
within it:
extrn_mdl_source_dir = \"${extrn_mdl_source_dir}\"
extrn_mdl_fns_on_disk = ( $( printf "\"%s\" " "${extrn_mdl_fns_on_disk[@]}" ))"
fi

fi
staging_dir="${CYCLE_DIR}/${EXTRN_MDL_NAME}/for_${ICS_OR_LBCS}"
mkdir_vrfy -p "${staging_dir}"
cd_vrfy "${staging_dir}"
#
#-----------------------------------------------------------------------
#
# Call the ex-script for this J-job and pass to it the necessary variables.
#
#-----------------------------------------------------------------------
#
extrn_mdl_lbc_spec_fhrs_str="( "$( printf "\"%s\" " "${extrn_mdl_lbc_spec_fhrs[@]}" )")"
extrn_mdl_fns_on_disk_str="( "$( printf "\"%s\" " "${extrn_mdl_fns_on_disk[@]}" )")"
extrn_mdl_fns_in_arcv_str="( "$( printf "\"%s\" " "${extrn_mdl_fns_in_arcv[@]}" )")"
extrn_mdl_arcv_fns_str="( "$( printf "\"%s\" " "${extrn_mdl_arcv_fns[@]}" )")"
extrn_mdl_arcv_fps_str="( "$( printf "\"%s\" " "${extrn_mdl_arcv_fps[@]}" )")"
extrn_mdl_data_sources_str="( "$( printf "\"%s\" " "${EXTRN_MDL_DATA_SOURCES[@]}" )")"

$SCRIPTSDIR/exregional_get_extrn_mdl_files.sh \
ics_or_lbcs="${ICS_OR_LBCS}" \
use_user_staged_extrn_files="${USE_USER_STAGED_EXTRN_FILES}" \
extrn_mdl_cdate="${extrn_mdl_cdate}" \
extrn_mdl_lbc_spec_fhrs="${extrn_mdl_lbc_spec_fhrs_str}" \
extrn_mdl_fns_on_disk="${extrn_mdl_fns_on_disk_str}" \
extrn_mdl_fns_in_arcv="${extrn_mdl_fns_in_arcv_str}" \
extrn_mdl_source_dir="${extrn_mdl_source_dir}" \
extrn_mdl_staging_dir="${extrn_mdl_staging_dir}" \
extrn_mdl_arcv_fmt="${extrn_mdl_arcv_fmt}" \
extrn_mdl_arcv_fns="${extrn_mdl_arcv_fns_str}" \
extrn_mdl_arcv_fps="${extrn_mdl_arcv_fps_str}" \
extrn_mdl_arcvrel_dir="${extrn_mdl_arcvrel_dir}" || \
data_sources="${extrn_mdl_data_sources_str}" \
staging_dir="${staging_dir}" || \
print_err_msg_exit "\
Call to ex-script corresponding to J-job \"${scrfunc_fn}\" failed."
#
Expand Down
2 changes: 1 addition & 1 deletion jobs/JREGIONAL_MAKE_ICS
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
#
#-----------------------------------------------------------------------
#
{ save_shell_opts; set -u -x; } > /dev/null 2>&1
{ save_shell_opts; set -u +x; } > /dev/null 2>&1
#
#-----------------------------------------------------------------------
#
Expand Down
2 changes: 1 addition & 1 deletion jobs/JREGIONAL_RUN_POST
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
#
#-----------------------------------------------------------------------
#
{ save_shell_opts; set -u -x; } > /dev/null 2>&1
{ save_shell_opts; set -u +x; } > /dev/null 2>&1
#
#-----------------------------------------------------------------------
#
Expand Down
Loading