Use stochastic restart patterns on rerun by WalterKolczynski-NOAA · Pull Request #3077 · NOAA-EMC/global-workflow

WalterKolczynski-NOAA · 2024-11-07T18:55:09Z

Description

The stochastic pattern restart files were not being copied into the input directory when restarting the model after a segment/failure. These files are now copied in and the stochini flag set to .true. in the namelist on a rerun.

The files are NOT copied in for non-rerun warm starts.

Also removes the restriction that stochastic physics cannot be run on member 0, as this is desired down the line. Additional settings are added to the fcst and efcs configs to retain the current behavior.

A bug was also discovered and corrected during this work. The stage, forecast, and archive job all assumed that ca_data tile files are always present, but these are only created when cellular automata is on. Now ca_data files are handled if CA is on. To implement this change, the DO_CA setting had to be moved from the forecast configs to base so it is available to the stage_ic and archive jobs.

Resolves #2937

Type of change

Bug fix (fixes something broken)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO

How has this been tested?

Multi-segment test on Hercules

Checklist

aerorahul

looks good to me. No comments.
Please invite review from @pjpegion @NeilBarton-NOAA

emcbot · 2024-11-13T11:46:00Z

Experiment C96_atm3DVar FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96_atm3DVar_d9209edc

emcbot · 2024-11-13T11:46:01Z

Experiment C48mx500_3DVarAOWCDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_d9209edc/logs/2021032412/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-11-13T11:46:07Z

Experiment C48mx500_3DVarAOWCDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_d9209edc

emcbot · 2024-11-13T11:52:07Z

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_d9209edc/logs/2024022318/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-11-13T11:52:08Z

Experiment C96C48_hybatmDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmDA_d9209edc/logs/2021122018/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-11-13T11:52:13Z

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_ufs_hybatmDA_d9209edc

emcbot · 2024-11-13T11:52:13Z

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_d9209edc/logs/2021122012/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-11-13T11:52:14Z

Experiment C96C48_hybatmDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_hybatmDA_d9209edc

emcbot · 2024-11-13T11:52:20Z

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_hybatmaerosnowDA_d9209edc

emcbot · 2024-11-13T15:38:10Z

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem000_seg1.log
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem001_seg1.log
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem002_seg1.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-11-13T15:38:19Z

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C48_S2SWA_gefs_d9209edc

emcbot · 2024-11-13T23:13:54Z

CI Failed on Hera in Build# 1
Built and ran in directory /scratch1/NCEPDEV/global/CI/3077


Experiment C48mx500_3DVarAOWCDA_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:45:45 UTC 2024
Experiment C48mx500_3DVarAOWCDA_d9209edc Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_d9209edc/logs/2021032412/gdas_fcst_seg0.log
Experiment C96_atm3DVar_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:45:50 UTC 2024
Experiment C96_atm3DVar_d9209edc Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96_atm3DVar_d9209edc/logs/2021122018/gdas_fcst_seg0.log
Experiment C96C48_hybatmDA_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:51:56 UTC 2024
Experiment C96C48_hybatmDA_d9209edc Terminated: *FAIL*
Experiment C96C48_ufs_hybatmDA_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:51:57 UTC 2024
Experiment C96C48_ufs_hybatmDA_d9209edc Terminated: *FAIL*
Experiment C96C48_hybatmaerosnowDA_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:51:59 UTC 2024
Experiment C96C48_hybatmaerosnowDA_d9209edc Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmDA_d9209edc/logs/2021122018/gdas_fcst_seg0.log
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_d9209edc/logs/2024022318/gdas_fcst_seg0.log
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_d9209edc/logs/2021122012/gdas_fcst_seg0.log
Experiment C48_S2SWA_gefs_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 3 dead at Wed Nov 13 15:37:41 UTC 2024
Experiment C48_S2SWA_gefs_d9209edc Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem000_seg1.log
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem001_seg1.log
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem002_seg1.log
Experiment C96_S2SWA_gefs_replay_ics_d9209edc Completed 1 Cycles: *SUCCESS* at Wed Nov 13 17:25:52 UTC 2024
Experiment C48_ATM_d9209edc Completed 2 Cycles: *SUCCESS* at Wed Nov 13 18:45:20 UTC 2024
Experiment C48_S2SW_d9209edc Completed 2 Cycles: *SUCCESS* at Wed Nov 13 23:13:39 UTC 2024

emcbot · 2024-12-03T07:59:54Z

CI Passed on Hercules in Build# 1
Built and ran in directory /work2/noaa/global/CI/HERCULES/3077


Experiment C48_ATM_745f4ade Completed 2 Cycles: *SUCCESS* at Mon Dec  2 21:56:28 CST 2024
Experiment C96_S2SWA_gefs_replay_ics_745f4ade Completed 1 Cycles: *SUCCESS* at Mon Dec  2 22:45:05 CST 2024
Experiment C96_atm3DVar_745f4ade Completed 3 Cycles: *SUCCESS* at Mon Dec  2 22:57:01 CST 2024
Experiment C96C48_hybatmDA_745f4ade Completed 3 Cycles: *SUCCESS* at Mon Dec  2 23:03:04 CST 2024
Experiment C48_S2SW_745f4ade Completed 2 Cycles: *SUCCESS* at Mon Dec  2 23:45:44 CST 2024
Experiment C48_S2SWA_gefs_745f4ade Completed 1 Cycles: *SUCCESS* at Tue Dec  3 01:59:40 CST 2024

WalterKolczynski-NOAA · 2024-12-03T17:15:51Z

CI Tests set up to run in /lfs/h2/emc/ptmp/walter.kolczynski/PR/PR_3077/RUNTESTS on WCOSS

WalterKolczynski-NOAA · 2024-12-04T14:13:16Z

Got all the way to the last cycle of the extended test and then died. Investigating.

WalterKolczynski-NOAA · 2024-12-04T14:14:15Z

Looks like stmp filled on WCOSS

emcbot · 2024-12-04T17:06:18Z

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 4 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_805c1648/logs/2021122018/enkfgdas_earc01.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-12-04T17:06:27Z

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 4 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_hybatmaerosnowDA_805c1648

emcbot · 2024-12-04T18:11:34Z

Experiment C96_atm3DVar FAILED on Hera in Build# 4 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96_atm3DVar_805c1648

emcbot · 2024-12-04T18:11:37Z

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 4 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_ufs_hybatmDA_805c1648

emcbot · 2024-12-04T18:11:38Z

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 4 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C48_S2SWA_gefs_805c1648

emcbot · 2024-12-04T18:11:41Z

Experiment C96C48_hybatmDA FAILED on Hera in Build# 4 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_hybatmDA_805c1648

emcbot · 2024-12-04T18:11:45Z

Experiment C48_S2SW FAILED on Hera in Build# 4 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C48_S2SW_805c1648

emcbot · 2024-12-04T18:11:54Z

CI Failed on Hera in Build# 4
Built and ran in directory /scratch1/NCEPDEV/global/CI/3077


Experiment C48mx500_3DVarAOWCDA_805c1648 Completed 2 Cycles: *SUCCESS* at Wed Dec  4 17:05:53 UTC 2024
Experiment C96C48_hybatmaerosnowDA_805c1648 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Dec  4 17:06:00 UTC 2024
Experiment C96C48_hybatmaerosnowDA_805c1648 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_805c1648/logs/2021122018/enkfgdas_earc01.log
Experiment C48_ATM_805c1648 Completed 2 Cycles: *SUCCESS* at Wed Dec  4 17:11:59 UTC 2024
Experiment C96_S2SWA_gefs_replay_ics_805c1648 Completed 1 Cycles: *SUCCESS* at Wed Dec  4 17:24:51 UTC 2024

emcbot · 2024-12-04T21:29:49Z

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 5 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_33615413/logs/2021122018/enkfgdas_earc01.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-12-04T21:29:56Z

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 5 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_hybatmaerosnowDA_33615413

emcbot · 2024-12-04T22:13:08Z

Experiment C48_S2SW FAILED on Hera in Build# 5 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C48_S2SW_33615413

emcbot · 2024-12-04T22:13:11Z

Experiment C96C48_hybatmDA FAILED on Hera in Build# 5 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_hybatmDA_33615413

emcbot · 2024-12-04T22:13:14Z

Experiment C96_atm3DVar FAILED on Hera in Build# 5 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96_atm3DVar_33615413

emcbot · 2024-12-04T22:13:14Z

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 5 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_ufs_hybatmDA_33615413

emcbot · 2024-12-04T22:13:18Z

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 5 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C48_S2SWA_gefs_33615413

emcbot · 2024-12-04T22:13:27Z

CI Failed on Hera in Build# 5
Built and ran in directory /scratch1/NCEPDEV/global/CI/3077


Experiment C48mx500_3DVarAOWCDA_33615413 Completed 2 Cycles: *SUCCESS* at Wed Dec  4 20:52:51 UTC 2024
Experiment C48_ATM_33615413 Completed 2 Cycles: *SUCCESS* at Wed Dec  4 21:11:03 UTC 2024
Experiment C96_S2SWA_gefs_replay_ics_33615413 Completed 1 Cycles: *SUCCESS* at Wed Dec  4 21:17:37 UTC 2024
Experiment C96C48_hybatmaerosnowDA_33615413 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Dec  4 21:29:38 UTC 2024
Experiment C96C48_hybatmaerosnowDA_33615413 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_33615413/logs/2021122018/enkfgdas_earc01.log

emcbot · 2024-12-05T06:45:16Z

CI Passed on Hera in Build# 6
Built and ran in directory /scratch1/NCEPDEV/global/CI/3077


Experiment C48_ATM_9ca245ef Completed 2 Cycles: *SUCCESS* at Thu Dec  5 00:51:54 UTC 2024
Experiment C96_S2SWA_gefs_replay_ics_9ca245ef Completed 1 Cycles: *SUCCESS* at Thu Dec  5 01:23:03 UTC 2024
Experiment C48_S2SW_9ca245ef Completed 2 Cycles: *SUCCESS* at Thu Dec  5 03:24:49 UTC 2024
Experiment C48_S2SWA_gefs_9ca245ef Completed 1 Cycles: *SUCCESS* at Thu Dec  5 03:56:47 UTC 2024
Experiment C48mx500_3DVarAOWCDA_9ca245ef Completed 2 Cycles: *SUCCESS* at Thu Dec  5 04:18:34 UTC 2024
Experiment C96C48_hybatmaerosnowDA_9ca245ef Completed 3 Cycles: *SUCCESS* at Thu Dec  5 05:38:22 UTC 2024
Experiment C96_atm3DVar_9ca245ef Completed 3 Cycles: *SUCCESS* at Thu Dec  5 05:44:20 UTC 2024
Experiment C96C48_hybatmDA_9ca245ef Completed 3 Cycles: *SUCCESS* at Thu Dec  5 05:44:26 UTC 2024
Experiment C96C48_ufs_hybatmDA_9ca245ef Completed 3 Cycles: *SUCCESS* at Thu Dec  5 06:45:05 UTC 2024

KateFriedman-NOAA

I can't comment on the science updates in this PR but the workflow changes look good. Thanks @WalterKolczynski-NOAA !

WalterKolczynski-NOAA requested review from NeilBarton-NOAA and aerorahul November 7, 2024 18:55

WalterKolczynski-NOAA requested a review from junwang-noaa as a code owner November 7, 2024 18:55

WalterKolczynski-NOAA requested a review from pjpegion November 7, 2024 20:39

aerorahul previously approved these changes Nov 12, 2024

View reviewed changes

WalterKolczynski-NOAA added the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Nov 13, 2024

NeilBarton-NOAA reviewed Nov 13, 2024

View reviewed changes

Comment thread ush/forecast_postdet.sh

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Nov 13, 2024

aerorahul requested a review from bingfu-NOAA November 14, 2024 19:42

bingfu-NOAA reviewed Nov 14, 2024

View reviewed changes

Comment thread parm/config/gefs/config.fcst Outdated

bingfu-NOAA reviewed Nov 14, 2024

View reviewed changes

Comment thread parm/config/gfs/config.fcst Outdated

DavidHuber-NOAA reviewed Dec 3, 2024

View reviewed changes

Comment thread parm/config/gefs/config.base

Merge branch 'develop' into feature/stoch_restart

4bca477

DavidHuber-NOAA previously approved these changes Dec 3, 2024

View reviewed changes

Merge branch 'develop' into feature/stoch_restart

805c164

Add DO_CA to earc keys

3361541

Fix earc path for ca_data

9ca245e

KateFriedman-NOAA approved these changes Dec 5, 2024

View reviewed changes

This was referenced Dec 10, 2024

Turn C96C48_ufs_hybatmDA and C48mx500_3DVarAOWCDA into a regression test #3120

Merged

Update select C96C48_ufs_hybatmDA reference files NOAA-EMC/GDASApp#1411

Closed

Conversation

WalterKolczynski-NOAA commented Nov 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

aerorahul left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emcbot commented Nov 13, 2024

Uh oh!

emcbot commented Nov 13, 2024

Uh oh!

emcbot commented Nov 13, 2024

Uh oh!

emcbot commented Nov 13, 2024

Uh oh!

emcbot commented Nov 13, 2024

Uh oh!

emcbot commented Nov 13, 2024

Uh oh!

emcbot commented Nov 13, 2024

Uh oh!

emcbot commented Nov 13, 2024

Uh oh!

emcbot commented Nov 13, 2024

Uh oh!

emcbot commented Nov 13, 2024

Uh oh!

emcbot commented Nov 13, 2024

Uh oh!

Uh oh!

emcbot commented Nov 13, 2024

Uh oh!

Uh oh!

Uh oh!

emcbot commented Dec 3, 2024

Uh oh!

Uh oh!

WalterKolczynski-NOAA commented Dec 3, 2024

Uh oh!

WalterKolczynski-NOAA commented Dec 4, 2024

Uh oh!

WalterKolczynski-NOAA commented Dec 4, 2024

Uh oh!

emcbot commented Dec 4, 2024

Uh oh!

emcbot commented Dec 4, 2024

Uh oh!

emcbot commented Dec 4, 2024

Uh oh!

emcbot commented Dec 4, 2024

Uh oh!

emcbot commented Dec 4, 2024

Uh oh!

emcbot commented Dec 4, 2024

Uh oh!

emcbot commented Dec 4, 2024

Uh oh!

emcbot commented Dec 4, 2024

Uh oh!

emcbot commented Dec 4, 2024

Uh oh!

emcbot commented Dec 4, 2024

Uh oh!

emcbot commented Dec 4, 2024

Uh oh!

emcbot commented Dec 4, 2024

Uh oh!

emcbot commented Dec 4, 2024

Uh oh!

emcbot commented Dec 4, 2024

Uh oh!

emcbot commented Dec 4, 2024

Uh oh!

emcbot commented Dec 4, 2024

WalterKolczynski-NOAA commented Nov 7, 2024 •

edited

Loading

aerorahul left a comment •

edited

Loading