Add task to process reforecast variables to save on WCOSS2#2680
Merged
WalterKolczynski-NOAA merged 157 commits intoJul 23, 2024
Merged
Add task to process reforecast variables to save on WCOSS2#2680WalterKolczynski-NOAA merged 157 commits into
WalterKolczynski-NOAA merged 157 commits into
Conversation
A rocoto task has been added to process the reforecast variables. A config file has been created for this task. Resources specific to this task have also been added to config.resources.
The topmost extractvars script has been added to the global workflow rocoto subdirectory.
The extractvars job dependency has been updated so that extractvars is triggered by the completion of wave_post_grid.
A new JGLOBAL script has been added for the extractvars task.
Scripts have been added which allow the processing of atmosphere reforecast variables.
Parm files have been added that provide a variable list specific to reforecast variables to be saved on disk.
The extractvars scripts have been simplified.
Corrections were made to some new scripts.
A script has been added to process the ocean and ice variables for the reforecast. Parm files for ocean and ice have also been added.
A script has been added to process the wave variables for the reforecast.
The script that processes waves in the extractvars task has been cleaned up.
The brackets have been fixed in gefs_tasks.py
A wave output frequency variable has been added that allows a specific wave forecast output frequency for data saved on disk.
Shellnorms is set to run on every git push.
An FHMAX variable has been added to the wave script in the extractvars task so that an unbound error can be avoided.
The JGLOBAL_EXTRACTVARS script has been refined.
More scripts have been cleaned up.
This reverts commit b79237c.
The extactvars script has been cleaned up.
emcbot
added
CI-Hera-Failed
aerorahul
reviewed
Jul 19, 2024
|
|
||
| if [[ -f "${infile}" ]]; then #check if input file exists before extraction | ||
| # shellcheck disable=SC2312 | ||
| ${WGRIB2} "${infile}" | grep -F -f "${varlist_wav}" | ${WGRIB2} -i "${infile}" -append -grib "${outfile}">/dev/null |
Contributor
There was a problem hiding this comment.
Should we be checking the status of this call? The output is being sent to /dev/null.
If there is any issue with this call, it will not be seen.
aerorahul
reviewed
Jul 19, 2024
| for infile in "${infile1}" "${infile2}"; do | ||
| if [[ -f "${infile}" ]]; then # check if input file exists before extraction | ||
| # shellcheck disable=SC2312 | ||
| ${WGRIB2} "${infile}" | grep -F -f "${varlist}" | ${WGRIB2} -i "${infile}" -append -grib "${outfile}" > /dev/null |
Contributor
There was a problem hiding this comment.
Same comment as before. Any issue in this call will be sent to /dev/null. It should at least be checked for a exit code 0 to ensure the command was successful before proceeding.
Contributor
|
The runner failed due to disk space restrictions while cloning. This appears to be a restriction on Jenkins, even though it is cloning to Hera. I will investigate this and apply a fix in #2780 before launching again. |
emcbot
removed
the
CI-Hercules-Running
|
Experiment C96_atm3DVar FAILED on Hercules in |
|
Experiment C96C48_hybatmDA FAILED on Hercules in |
aerorahul
removed
CI-Hera-Failed
DavidHuber-NOAA
added
the
CI-Hercules-Ready
emcbot
added
CI-Hercules-Building
|
CI Passed Hercules at |
WalterKolczynski-NOAA
approved these changes
Jul 23, 2024
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds an optional task to the global-workflow to process a subset of ocean, ice, wave and atmosphere products to be saved on WCOSS2 for the GEFSv13 reforecast. This task is designed to process GEFS variables so that specific reforecast product requirements are met. A new variable in
config.basecalledDO_EXTRACTVARSenables this task, which is currently calledextractvars.DO_EXTRACTVARSis set toNOby default and is specifically a task designed to be executed for the GEFSv13 reforecast.Refs #1878
Type of change
Change characteristics
How has this been tested?
This has been cloned and tested on WCOSS2. This will need to be tested on Hera and other platforms on which the reforecast may be running.
Checklist