[develop] Consolidate verification tasks using retrieve_data.py#864

Merged

MichaelLueken merged 24 commits into

ufs-community:developfrom

mkavulich:feature/issue_850

Aug 15, 2023

Commits on Aug 14, 2023

Remove unnecessary "minus 1" dates
mkavulich
committed
Remove unused logic for 3- and 6-hour accumulation times
mkavulich
committed
Convert exregional_get_obs_ccpa.sh to use retrieve_data.py...this version may have bugs, I am committing so I can test on another platform since my Jet "fairshare" is apparently off-the-charts low
mkavulich
committed
New script working for CCPA obs 🥳
mkavulich
committed
Fix bug in retrieve_data.py: when the glob expression for files in the tarball contains a full path, we need to lstrip the leading "/" before globbing
mkavulich
committed
Consolidate all "get_obs" tasks for vx into single J-job/exscript.
mkavulich
committed
Now that retrieve_data.py checks for the existence of the output directory, we need to create it prior to calling the script
mkavulich
committed
Complete transition to new obs pulling script. Currently works to pull
mkavulich
committed
Get rid of unneccessary directory structure requirement for obs. All verification tests now pass when retrieving from HPSS!
mkavulich
committed
Wildcards not allowed in 'archive_file_names' entry
mkavulich
committed
Updates to exregional_get_verif_obs.sh:
mkavulich
committed
Updates to verification tests:
mkavulich
committed
More testing changes/updates
mkavulich
committed
Fix write component for new test
mkavulich
committed
Replace "#!/bin/bash" with "#!/usr/bin/env bash" per Issue 857
mkavulich
committed
Update some comments and prints for new j-Job
mkavulich
committed
Make script mrms_pull_topofhour.py raise an exception if no file is found
mkavulich
committed
Remove immediate quit when specified output path does not exist; this didn't account for templating behavior
mkavulich
committed
Fix error in output directory name for test "test_gdas_ics_from_aws"
mkavulich
committed
Roll new snowfall/NOHRSC verification into new data retrieval system
mkavulich
committed
Cheyenne can't automatically get AWS data, removing 2020_CAD test from comprehensive.cheyenne
mkavulich
committed
Address Gerard's comments
mkavulich
committed
Move machine file definitions of *_OBS_DIRs to TEST_*_OBS_DIR for WE2E testing purposes, similar to TEST_ALT_EXTRN_MDL_SYSBASEDIR_ICS
mkavulich
committed

Commits on Aug 15, 2023

Temporarily remove HPSS-pulling verification tests from coverage suites
mkavulich
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[develop] Consolidate verification tasks using retrieve_data.py#864

[develop] Consolidate verification tasks using retrieve_data.py#864
MichaelLueken merged 24 commits into
ufs-community:developfrom
mkavulich:feature/issue_850

Commits on Aug 14, 2023

Remove unnecessary "minus 1" dates

Remove unused logic for 3- and 6-hour accumulation times

Convert exregional_get_obs_ccpa.sh to use retrieve_data.py...this version may have bugs, I am committing so I can test on another platform since my Jet "fairshare" is apparently off-the-charts low

New script working for CCPA obs 🥳

Fix bug in retrieve_data.py: when the glob expression for files in the tarball contains a full path, we need to lstrip the leading "/" before globbing

Consolidate all "get_obs" tasks for vx into single J-job/exscript.

Now that retrieve_data.py checks for the existence of the output directory, we need to create it prior to calling the script

Complete transition to new obs pulling script. Currently works to pull

Get rid of unneccessary directory structure requirement for obs. All verification tests now pass when retrieving from HPSS!

Wildcards not allowed in 'archive_file_names' entry

Updates to exregional_get_verif_obs.sh:

Updates to verification tests:

More testing changes/updates

Fix write component for new test

Replace "#!/bin/bash" with "#!/usr/bin/env bash" per Issue 857

Update some comments and prints for new j-Job

Make script mrms_pull_topofhour.py raise an exception if no file is found

Remove immediate quit when specified output path does not exist; this didn't account for templating behavior

Fix error in output directory name for test "test_gdas_ics_from_aws"

Roll new snowfall/NOHRSC verification into new data retrieval system

Cheyenne can't automatically get AWS data, removing 2020_CAD test from comprehensive.cheyenne

Address Gerard's comments

Move machine file definitions of _OBS_DIRs to TEST__OBS_DIR for WE2E testing purposes, similar to TEST_ALT_EXTRN_MDL_SYSBASEDIR_ICS

Commits on Aug 15, 2023

Temporarily remove HPSS-pulling verification tests from coverage suites