Rename wave output and refactor some wave scripts to use MPMD, and fix some bugzillas along the way#3517
Conversation
…cast renaming of the output
…o forecast renaming of the output
aerorahul
changed the title
|
Please note that #3450 PR is a major update of the point jobs. Apologies I know I talked to Walter about not updating that job until after this major update, I did not properly pass this message on to you @aerorahul. This PR is waiting for the last sync from develop -> dev/ufs-waether-model for WW3 in UFS. This PR would have already been submitted but jobs failed yesterday w/stmp being full on hera. I'll try to get it submitted today, but not sure how long it'll take to get that through. |
Thank you for the note @JessicaMeixner-NOAA |
|
This is now ready for review |
aerorahul
requested review from
DavidHuber-NOAA,
JessicaMeixner-NOAA,
KateFriedman-NOAA,
WalterKolczynski-NOAA,
WenMeng-NOAA and
sbanihash
as code owners
|
I think this is now done, and I have checked the arch job runs past the failure point (I haven't had it succeed on Gaea because of access to HPSS [ticket submitted]) |
aerorahul
added
CI-Hera-Ready
emcbot
added
CI-Hera-Building
|
Experiment C96C48_ufs_hybatmDA FAILED on Gaeac6 in Build# 6 with error logs: Follow link here to view the contents of the above file(s): (link) |
|
Experiment C96C48_ufs_hybatmDA FAILED on Gaeac6 in Build# 6 in |
|
Experiment C48_S2SW FAILED on Hera in Build# 7 in |
|
Experiment C48_S2SWA_gefs FAILED on Hera in Build# 7 in |
|
Experiment C96mx100_S2S FAILED on Hera in Build# 7 in |
|
Experiment C48_ATM FAILED on Hera in Build# 7 in |
|
Experiment C48mx500_hybAOWCDA FAILED on Hera in Build# 7 in |
|
Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 7 in |
|
Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 7 in |
|
Experiment C48mx500_3DVarAOWCDA FAILED on Hera in Build# 7 in |
|
Experiment C96_atm3DVar FAILED on Hera in Build# 7 in |
|
Experiment C96C48_hybatmDA FAILED on Hera in Build# 7 in |
|
CI Failed on Hera in Build# 7 This failure was due to resources limitations and/or anomalies from within shell for the system to able to create enough processes to fork when running |
|
CI Failed on Gaeac6 in Build# 6 |
…ccessfully and the logs are not helpful in diagnosing the error
@RussTreadon-NOAA @DavidNew-NOAA |
|
Changing the |
|
All the Hera tests from build number 7 have been restarted where they left off from the above initial failure in Note: the |
6 participants
Description
[Copied and edited from #3406]
Wave products are renamed to be uniform with other components and follow NCO implementation standards. This means all output is in the format ${RUN}.wave.tCCz.product[.fHHH][.domain].suffix.
Some wave scripts are refactored to make them consistent with other parts of the workflow. All scripts for those jobs now pass shellcheck without warnings. Point job scripts were only updated for the file rename, as those scripts will be redone in the near future anyway.
The biggest changes in the refactor include beyond those necessary for the file rename include:
Using the run_mpmd.sh script
Removed redirection of output to logs that get deleted (#296)
The wave gempak job now only operates on one forecast hour at a time. The corresponding rocoto task has been made into a metatask, using the same grouping as wave gridded post. The gempak job must use the same fhr and max tasks settings to ensure the dependencies line up.
One functional change is raw interpolated grids are no longer copied to COM, only the resulting grib files. [Need to confirm with Jessica this is okay]
Resolves #296
Resolves #3270
This was work was largely done by @WalterKolczynski-NOAA
Type of change
Change characteristics
How has this been tested?
A experiment with
developat 563567ddev1at:/lfs/h2/emc/ptmp/rahul.mahajan/RUNTESTS/EXPDIR/dev1A experiment with this branch
wk1at:/lfs/h2/emc/ptmp/rahul.mahajan/RUNTESTS/EXPDIR/wk1Links to respective
COMcan be found in the experiment directories.Below is a list of wave products from
wk1❯❯❯ tree COM/gfs.20210323/12/products/wave/ COM/gfs.20210323/12/products/wave/ ├── gempak │ ├── gfswaves200k_2021032312f000 │ ├── gfswaves200k_2021032312f001 │ ├── ... │ ├── ... │ └── gfswaves200k_2021032312f120 ├── gridded │ └── global.2p00 │ ├── gfs.wave.t12z.global.2p00.f000.grib2 │ ├── gfs.wave.t12z.global.2p00.f000.grib2.idx │ ├── ... │ ├── ... │ ├── gfs.wave.t12z.global.2p00.f120.grib2 │ └── gfs.wave.t12z.global.2p00.f120.grib2.idx └── station ├── gfs.wave.t12z.t12z.bull_tar ├── gfs.wave.t12z.t12z.cbull_tar ├── gfs.wave.t12z.t12z.ibpbull_tar ├── gfs.wave.t12z.t12z.ibpcbull_tar ├── gfs.wave.t12z.t12z.ibp_tar └── gfs.wave.t12z.t12z.spec_tar.gz 4 directories, 369 filesThe forecast model produced wave model output as:
❯❯❯ tree COM/gfs.20210323/12/model/wave/ ├── history │ ├── gfs.wave.t12z.log.uglo_100km.2021032312 │ ├── gfs.wave.t12z.points.f000.bin │ ├── gfs.wave.t12z.points.f001.bin │ ├── ... │ ├── gfs.wave.t12z.points.f120.bin │ ├── gfs.wave.t12z.uglo_100km.f000.bin │ ├── ... │ ├── ... │ ├── ... │ └── gfs.wave.t12z.uglo_100km.f120.bin ├── prep │ ├── gfs.wave.t12z.mod_def.glo_200.bin │ └── gfs.wave.t12z.mod_def.uglo_100km.bin └── restart 3 directories, 245 filesChecklist