Skip to content

[develop] Consolidate verification tasks using retrieve_data.py #864

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
MichaelLueken merged 24 commits into from
Aug 15, 2023
Merged

[develop] Consolidate verification tasks using retrieve_data.py #864

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
9070063
Remove unnecessary "minus 1" dates
mkavulich Jul 11, 2023
faf5f72
Remove unused logic for 3- and 6-hour accumulation times
mkavulich Jul 12, 2023
1a5da5d
Convert exregional_get_obs_ccpa.sh to use retrieve_data.py...this ver…
mkavulich Jul 12, 2023
b383c3f
New script working for CCPA obs 🥳
mkavulich Jul 12, 2023
93a6ca4
Fix bug in retrieve_data.py: when the glob expression for files in th…
mkavulich Jul 12, 2023
77fcc4c
Consolidate all "get_obs" tasks for vx into single J-job/exscript.
mkavulich Jul 13, 2023
5230d7a
Now that retrieve_data.py checks for the existence of the output dire…
mkavulich Jul 17, 2023
58ef36f
Complete transition to new obs pulling script. Currently works to pull
mkavulich Jul 17, 2023
b199379
Get rid of unneccessary directory structure requirement for obs. All …
mkavulich Jul 18, 2023
906846b
Wildcards not allowed in 'archive_file_names' entry
mkavulich Jul 18, 2023
e998ded
Updates to exregional_get_verif_obs.sh:
mkavulich Jul 18, 2023
d4cb266
Updates to verification tests:
mkavulich Jul 18, 2023
b666879
More testing changes/updates
mkavulich Jul 25, 2023
2882610
Fix write component for new test
mkavulich Jul 19, 2023
9cfbcd1
Replace "#!/bin/bash" with "#!/usr/bin/env bash" per Issue 857
mkavulich Jul 19, 2023
712207e
Update some comments and prints for new j-Job
mkavulich Jul 19, 2023
48fb9d3
Make script mrms_pull_topofhour.py raise an exception if no file is f…
mkavulich Jul 19, 2023
2dbdf94
Remove immediate quit when specified output path does not exist; this…
mkavulich Jul 19, 2023
ef161a2
Fix error in output directory name for test "test_gdas_ics_from_aws"
mkavulich Jul 19, 2023
0fde1cb
Roll new snowfall/NOHRSC verification into new data retrieval system
mkavulich Jul 20, 2023
b39decf
Cheyenne can't automatically get AWS data, removing 2020_CAD test fro…
mkavulich Jul 27, 2023
e35d28d
Address Gerard's comments
mkavulich Aug 4, 2023
624084d
Move machine file definitions of *_OBS_DIRs to TEST_*_OBS_DIR for WE2…
mkavulich Aug 4, 2023
6b97050
Temporarily remove HPSS-pulling verification tests from coverage suites
mkavulich Aug 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 0 additions & 91 deletions jobs/JREGIONAL_GET_OBS_CCPA
View file
Open in desktop

This file was deleted.

92 changes: 0 additions & 92 deletions jobs/JREGIONAL_GET_OBS_MRMS
View file
Open in desktop

This file was deleted.

91 changes: 0 additions & 91 deletions jobs/JREGIONAL_GET_OBS_NOHRSC
View file
Open in desktop

This file was deleted.

10 changes: 5 additions & 5 deletions jobs/JREGIONAL_GET_OBS_NDAS → jobs/JREGIONAL_GET_VERIF_OBS
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
#!/bin/bash
#!/usr/bin/env bash

#
#-----------------------------------------------------------------------
#
# This script pulls NDAS observation data for comparison to the model.
# This script checks, pulls, and stages observation data for model verification.
#
#-----------------------------------------------------------------------
#
Expand Down Expand Up @@ -51,8 +51,8 @@ print_info_msg "
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"

This is the J-job script for the task that pulls NDAS observation data
for verification purposes.
This is the J-job script for the task that checks, pulls, and stages
observation data for verification purposes.
========================================================================"

#
Expand All @@ -64,7 +64,7 @@ for verification purposes.
#
#-----------------------------------------------------------------------
#
$SCRIPTSdir/exregional_get_obs_ndas.sh || \
$SCRIPTSdir/exregional_get_verif_obs.sh || \
print_err_msg_exit "\
Call to ex-script corresponding to J-job \"${scrfunc_fn}\" failed."
#
Expand Down
Loading