Replace gfs_cyc with an interval#2928
Merged
WalterKolczynski-NOAA merged 31 commits intoOct 22, 2024
Merged
Conversation
WalterKolczynski-NOAA
requested review from
DavidHuber-NOAA,
KateFriedman-NOAA,
NeilBarton-NOAA,
aerorahul and
malloryprow
Contributor
Author
|
More manual testing to come, but ready enough to get eyes on it. |
WalterKolczynski-NOAA
force-pushed
the
feature/interval
branch
from
a532f64 to
f6141f8
Compare
7 tasks
WalterKolczynski-NOAA
force-pushed
the
feature/interval
branch
2 times, most recently
from
61d05ae to
1e92c40
Compare
To facilitate longer and more flexible GFS cadences, the `gfs_cyc` variable is replaced with a specified interval. Up front, this is reflected in a change in the arguments for setup_exp to: ``` --interval <n_hours> ``` Where `n_hours` is the interval (in hours) between gfs forecasts. `n_hours` must be a multiple of 6. If 0, no gfs will be run (only gdas; only valid for cycled mode). The default value is 6 (every cycle). In cycled mode, there is an additional argument to control which cycle will be the first gfs cycle: ``` ---sdate_gfs <YYYYMMDDHH> ``` The default if not provided is `--idate` + 6h (first full cycle). As part of this change, some of the validation of the dates has been added. `--edate` has also been made optional and defaults to `--idate` if not provided. During `config.base` template-filling, `INTERVAL_GFS` (renamed from `STEP_GFS`) is defined as `--interval` and `SDATE_GFS as `--sdate_gfs`. Some changes were necessary to the gfs verification (metp) job, as `gfs_cyc` was being used downstream by verif-global. That has been removed, and instead workflow will be responsible for only running metp on the correct cycles. This also removes "do nothing" metp tasks that exit immediately, because only the last GFS cycle in a day would actually process verification. Now, metp has its own cycledef and always runs at 18z, regardless of whether gfs is running at 18z or not. This is simplier than trying to determine the last gfs cycle of a day when it could change from day to day. To facilitate this change, support for the undocumented rocoto dependency tag `taskvalid` is added, as the metp task needs to know whether the cycle has a gfsarch task or not. metp will trigger on gfsarch completing (as before), or gdasarch completing if there is no gfsarch. metp tasks are no longer generated for forecast-only, as the pgbanl files (copied of the 1p00 pgbanl files) are not generated for f-o anyway. If metp is needed for f-o, additional work will be needed. Additionally, a couple EE2 issues with the metp job are resolved (even though it is not run in ops): - verif-global update replaced `$CDUMP` with `$RUN` - `$DATAROOT` is no longer redefined in the metp job Depends on NOAA-EMC/EMC_verif-global#137 Resolves NOAA-EMC#260 Refs NOAA-EMC#1299
To avoid running metp on days where there is no gfs, the cycledefs are adjusted somewhat. First, if the interval is >= 24h, the metp cycledef will be identical to gfs. If the interval is < 24h, it remains 18z every day (except the last). Second, a last_gfs is added so metp will run on for the last gfs cycle even if it there is no gdas cycle for 18z that day. This required computing the real gfs end date to use as the last cycle.
WalterKolczynski-NOAA
force-pushed
the
feature/interval
branch
from
18f9752 to
bfc2907
Compare
WalterKolczynski-NOAA
added
the
CI-Hercules-Ready
emcbot
added
CI-Hercules-Building
Contributor
Author
|
Ready for review. Also encourage third-party testing in excess of what CI will cover due to the nature of the change. I ran some different multi-day experiments myself, but outside validation would be appreciated. |
|
Automated global-workflow Testing Results: |
DavidHuber-NOAA
previously approved these changes
Oct 16, 2024
Contributor
DavidHuber-NOAA
left a comment
DavidHuber-NOAA
left a comment
There was a problem hiding this comment.
Looks good. Approve pending successful WCOSS2 testing.
DavidHuber-NOAA
previously approved these changes
Oct 16, 2024
Collaborator
|
Starting over from scratch with Walter's |
|
|
Automated global-workflow Testing Results: |
|
Experiment C96C48_hybatmaerosnowDA_d54bc650 FAIL on Wcoss2 at 10/16/24 09:14:32 PM Error logs: Follow link here to view the contents of the above file(s): (link) |
12 tasks
Contributor
Author
|
Sync'd missing snow observation data to cactus. |
The UFS model executable is not built for the structured wave grid used by GEFS on WCOSS, so GEFS tests should not run on WCOSS.
|
|
Automated global-workflow Testing Results: |
|
Experiment C96C48_hybatmaerosnowDA_2b6c45de FAIL on Wcoss2 at 10/17/24 06:56:28 PM Error logs: Follow link here to view the contents of the above file(s): (link) |
|
|
Automated global-workflow Testing Results: |
|
Experiment C96_atm3DVar_extended_2b6c45de FAIL on Wcoss2 at 10/18/24 08:35:52 PM Error logs: Follow link here to view the contents of the above file(s): (link) |
Contributor
Author
|
Failed because restarts from previous attempt weren't wiped, so forecast picked up in the middle. |
7 tasks
|
|
Automated global-workflow Testing Results: |
Contributor
Author
|
All manual CI tests passed on WCOSS: |
DavidHuber-NOAA
approved these changes
Oct 22, 2024
Contributor
DavidHuber-NOAA
left a comment
DavidHuber-NOAA
left a comment
There was a problem hiding this comment.
Thanks for all the work on this @WalterKolczynski-NOAA!
KateFriedman-NOAA
approved these changes
Oct 22, 2024
Contributor
KateFriedman-NOAA
left a comment
KateFriedman-NOAA
left a comment
There was a problem hiding this comment.
Looks good, thanks @WalterKolczynski-NOAA !
|
|
||
| date2 = sdate_gfs + interval_gfs | ||
| if date2 <= edate_gfs: | ||
| date2_gfs_str = date2_gfs.strftime("%Y%m%d%H%M") |
Contributor
There was a problem hiding this comment.
Just noticed that date2_gfs should be date2. I'll fix this in #2943.
Contributor
There was a problem hiding this comment.
I'm surprised PEP8 didn't complain about this.
38 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
To facilitate longer and more flexible GFS cadences, the
gfs_cycvariable is replaced with a specified interval. Up front, this isreflected in a change in the arguments for setup_exp to:
Where
n_hoursis the interval (in hours) between gfs forecasts.n_hoursmust be a multiple of 6. If 0, no gfs will be run (onlygdas; only valid for cycled mode). The default value is 6 (every cycle). (This is a change from current behavior of 24.)
In cycled mode, there is an additional argument to control which cycle will be the first gfs cycle:
The default if not provided is
--idate+ 6h (first full cycle). This is the same as current behavior whengfs_cycis 6, but may vary from current behavior for other cadences.As part of this change, some of the validation of the dates has been added.
--edatehas also been made optional and defaults to--idateif not provided.During
config.basetemplate-filling,INTERVAL_GFS(renamed fromSTEP_GFS) is defined as--intervalandSDATE_GFS as--sdate_gfs`.Some changes were necessary to the gfs verification (metp) job, as
gfs_cycwas being used downstream by verif-global. That has been removed, and instead workflow will be responsible for only running metp on the correct cycles. This also removes "do nothing" metp tasks that exit immediately, because only the last GFS cycle in a day would actually process verification.Now, metp has its own cycledef and will (a) always runs at 18z, regardless of whether gfs is running at 18z or not, if the interval is less than 24h; (b) use the same cycledef as gfs if the interval is 24h or greater. This is simpler than trying to determine the last gfs cycle of a day when it could change from day to day. To facilitate this change, support for the
undocumented rocoto dependency tag
taskvalidis added, as the metp task needs to know whether the cycle has a gfsarch task or not. metp will trigger on gfsarch completing (as before), or look backwards for the last gfsarch to exist.Additionally, a couple EE2 issues with the metp job are resolved (even though it is not run in ops):
$CDUMPwith$RUN$DATAROOTis no longer redefined in the metp jobDepends on NOAA-EMC/EMC_verif-global#137
Resolves #260
Refs #1299
Type of change
Change characteristics
How has this been tested?
Checklist