Adding a CI for using Replay ICs with an offset hour of 3 by NeilBarton-NOAA · Pull Request #2788 · NOAA-EMC/global-workflow

NeilBarton-NOAA · 2024-07-24T15:19:27Z

Description

This PR adds a CI testing using C96mx100 resolution and the S2SWA app. 2 perturb members are included with the control member.

Currently, this PR is dependent on
#2778
#2755

as the above PRs ensure the product tasks are being triggered in the workflow.

I could likely bypass the dependency #2755 if desired.

I will keep this PR as a draft until further discussion on the dependencies.

Current test ICs are located on hera at
/scratch2/NCEPDEV/stmp3/Neil.Barton/ICs/REPLAY_ICs/CI/2020110100

Type of change

New feature (adds functionality)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO

How has this been tested?

This is currently being tested on HERA.

Forecast-only on Hera

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
I have made corresponding changes to the documentation if necessary

changes. Lines starting

…lobal-workflow into gefs_replay_ci

NeilBarton-NOAA · 2024-08-14T13:36:17Z

ready for review. This will have a conflict with @KateFriedman-NOAA 's PR #2651

Fix a bug in a variable check in the extractvars task where parenthesis in the variable name will prompt a false warning message.

Ice variables are extracted from the history files instead of the product files. This is necessary because there are certain ice variables that are needed that are only available in the history files and not the product files.

emcbot · 2024-10-02T20:41:08Z

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 6 in
/scratch1/NCEPDEV/global/CI/2788/RUNTESTS/EXPDIR/C48_S2SWA_gefs_f918cc62

WalterKolczynski-NOAA · 2024-10-02T22:11:13Z

Looks like a 'real' error this time:

�[38;21m2024-10-02 20:37:31,785 - INFO     - oceanice_products: Copy ocean data to run directory�[0m
Traceback (most recent call last):
  File "/scratch1/NCEPDEV/global/CI/2788/gefs/ush/python/wxflow/fsutils.py", line 85, in cp
    shutil.copy2(source, target)
  File "/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/gsi-addon-dev-rocky8/install/intel/2021.5.0/python-3.11.6-b6ydksr/lib/python3.11/shutil.py", line 436, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/gsi-addon-dev-rocky8/install/intel/2021.5.0/python-3.11.6-b6ydksr/lib/python3.11/shutil.py", line 256, in copyfile
    with open(src, 'rb') as fsrc:
         ^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/scratch1/NCEPDEV/global/CI/2788/RUNTESTS/COMROOT/C48_S2SWA_gefs_f918cc62/gefs.20210323/12/mem000/model/ocean/history/gefs.ocean.t12z.6hr_avg.f006.nc'

emcbot · 2024-10-03T07:34:40Z

CI Failed on Hera in Build# 6
Built and ran in directory /scratch1/NCEPDEV/global/CI/2788


Experiment C48_S2SWA_gefs_f918cc62 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Oct  2 20:40:43 UTC 2024
Experiment C48_S2SWA_gefs_f918cc62 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/2788/RUNTESTS/COMROOT/C48_S2SWA_gefs_f918cc62/logs/2021032312/ocean_prod_mem000_f006.log
Experiment C48_ATM_f918cc62 Completed 1 Cycles: *SUCCESS* at Wed Oct  2 23:11:31 UTC 2024
Experiment C48mx500_3DVarAOWCDA_f918cc62 Completed 2 Cycles: *SUCCESS* at Wed Oct  2 23:24:04 UTC 2024
Experiment C48_S2SW_f918cc62 Completed 1 Cycles: *SUCCESS* at Wed Oct  2 23:25:49 UTC 2024
Experiment C96_atm3DVar_f918cc62 Completed 3 Cycles: *SUCCESS* at Thu Oct  3 01:03:03 UTC 2024
Experiment C96_S2SWA_gefs_replay_ics_f918cc62 Completed 1 Cycles: *SUCCESS* at Thu Oct  3 01:15:49 UTC 2024
Experiment C96C48_hybatmDA_f918cc62 Completed 3 Cycles: *SUCCESS* at Thu Oct  3 01:28:01 UTC 2024
Experiment C96C48_ufs_hybatmDA_f918cc62 Completed 2 Cycles: *SUCCESS* at Thu Oct  3 01:34:23 UTC 2024
Experiment C96C48_hybatmaerosnowDA_f918cc62 Completed 3 Cycles: *SUCCESS* at Thu Oct  3 07:34:26 UTC 2024

NeilBarton-NOAA · 2024-10-03T18:14:10Z

CI test fix updated. Please put through CI tests again

emcbot · 2024-10-05T10:46:19Z

CI Passed on Hera in Build# 7
Built and ran in directory /scratch1/NCEPDEV/global/CI/2788


Experiment C96_S2SWA_gefs_replay_ics_922eadff Completed 1 Cycles: *SUCCESS* at Sat Oct  5 03:55:04 UTC 2024
Experiment C48_ATM_922eadff Completed 1 Cycles: *SUCCESS* at Sat Oct  5 03:58:13 UTC 2024
Experiment C48_S2SW_922eadff Completed 1 Cycles: *SUCCESS* at Sat Oct  5 05:15:57 UTC 2024
Experiment C48mx500_3DVarAOWCDA_922eadff Completed 2 Cycles: *SUCCESS* at Sat Oct  5 08:38:15 UTC 2024
Experiment C96_atm3DVar_922eadff Completed 3 Cycles: *SUCCESS* at Sat Oct  5 09:47:09 UTC 2024
Experiment C48_S2SWA_gefs_922eadff Completed 1 Cycles: *SUCCESS* at Sat Oct  5 09:52:54 UTC 2024
Experiment C96C48_ufs_hybatmDA_922eadff Completed 2 Cycles: *SUCCESS* at Sat Oct  5 09:59:28 UTC 2024
Experiment C96C48_hybatmDA_922eadff Completed 3 Cycles: *SUCCESS* at Sat Oct  5 10:24:11 UTC 2024
Experiment C96C48_hybatmaerosnowDA_922eadff Completed 3 Cycles: *SUCCESS* at Sat Oct  5 10:46:08 UTC 2024

emcbot · 2024-10-07T20:22:58Z

CI Passed on Hercules in Build# 8
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2788


Experiment C48_ATM_922eadff Completed 1 Cycles: *SUCCESS* at Mon Oct  7 12:51:11 CDT 2024
Experiment C96_S2SWA_gefs_replay_ics_922eadff Completed 1 Cycles: *SUCCESS* at Mon Oct  7 12:57:19 CDT 2024
Experiment C96C48_hybatmDA_922eadff Completed 3 Cycles: *SUCCESS* at Mon Oct  7 13:51:59 CDT 2024
Experiment C96_atm3DVar_922eadff Completed 3 Cycles: *SUCCESS* at Mon Oct  7 13:57:47 CDT 2024
Experiment C48_S2SWA_gefs_922eadff Completed 1 Cycles: *SUCCESS* at Mon Oct  7 14:59:19 CDT 2024
Experiment C48_S2SW_922eadff Completed 1 Cycles: *SUCCESS* at Mon Oct  7 15:22:48 CDT 2024

KateFriedman-NOAA · 2024-10-23T17:57:36Z

@NeilBarton-NOAA I've pulled the updated IC set into ICSDIR on Hera. See here: /scratch1/NCEPDEV/global/glopara/data/ICSDIR/C96mx100/20240610

I pulled the files through the symlinks I saw within your set (rsync -azvL), so as to not have any symlinks to personal files. I also placed it under a timestamp folder. Let me know if the ICs look good on Hera and we'll sync them to the other platforms. After we sync them I will also remove the prior set that I pulled in that wasn't yet under the timestamp folder (/scratch1/NCEPDEV/global/glopara/data/ICSDIR/C96mx100/gefs.20201101).

@NeilBarton-NOAA Checking back on this. Please confirm the above ICs are good-to-go. Thanks!

NeilBarton-NOAA · 2024-10-23T18:37:20Z

@KateFriedman-NOAA The CI tests passed and the code has been merged. The ICs are good-to-go

KateFriedman-NOAA · 2024-10-23T18:39:55Z

Thanks for confirming @NeilBarton-NOAA , particularly for our records. I will proceed with deleting the non-timestamped copy (scratch1/NCEPDEV/global/glopara/data/ICSDIR/C96mx100/gefs.20201101).

NeilBarton-NOAA · 2024-10-23T19:11:57Z

@KateFriedman-NOAA I miss understood your question. The CI points to the timestamped ICs

https://github.com/NOAA-EMC/global-workflow/blob/develop/ci/cases/pr/C96_S2SWA_gefs_replay_ics.yaml

If removing the time-stamp folder, the yaml file will need to be edited.

KateFriedman-NOAA · 2024-10-23T19:17:14Z

@NeilBarton-NOAA All good, I removed the non-timestamped one. The timestamped one will remain and be used in the CI test.

NeilBarton-NOAA added 6 commits July 2, 2024 15:04

initial creation of CI using replay ICs

30cf5df

change name

4b3ae39

making an S2SWA case

fe2054e

Merge remote-tracking branch 'origin/develop' into gefs_replay_ci

7b7ee9b

Merge remote-tracking branch 'origin/develop' into gefs_replay_ci

225d280

start of CI for GEFS replays # Please enter the commit message for your

b17765e

changes. Lines starting

github-advanced-security AI found potential problems Jul 24, 2024

View reviewed changes

Comment thread jobs/JGLOBAL_WAVE_POST_PNT Fixed

Comment thread jobs/JGLOBAL_WAVE_POST_PNT Fixed

Comment thread parm/config/gefs/config.base Fixed

Comment thread parm/config/gefs/config.base Fixed

NeilBarton-NOAA and others added 7 commits July 24, 2024 15:25

Merge branch 'develop' into gefs_replay_ci

675cb77

removing shell check errors

e49dbcf

Merge branch 'gefs_replay_ci' of https://github.com/NeilBarton-NOAA/g…

ec4becc

…lobal-workflow into gefs_replay_ci

fixes for wave post processing

b69d653

Merge branch 'NOAA-EMC:develop' into gefs_replay_ci

084d744

Merge remote-tracking branch 'origin/develop' into gefs_replay_ci

a4bb8f3

add BREAKPOINTS variable

4ff31fd

EricSinsky-NOAA mentioned this pull request Aug 14, 2024

Add fixes to products for when REPLAY IC's are used #2755

Merged

7 tasks

NeilBarton-NOAA marked this pull request as ready for review August 14, 2024 13:34

NeilBarton-NOAA requested review from WalterKolczynski-NOAA and aerorahul August 14, 2024 13:34

fix shell checks

97b92ee

KateFriedman-NOAA reviewed Aug 14, 2024

View reviewed changes

Comment thread parm/config/gefs/config.ufs Outdated

EricSinsky-NOAA reviewed Aug 14, 2024

View reviewed changes

Comment thread ci/cases/yamls/gefs_replay_ci.yaml Outdated

EricSinsky-NOAA reviewed Aug 14, 2024

View reviewed changes

Comment thread ci/cases/yamls/gefs_replay_ci.yaml

NeilBarton-NOAA and others added 7 commits August 14, 2024 20:45

edit defult config.efc to be close to re-forecasting

713f41d

Fix bug in variable check

af31d2f

Fix a bug in a variable check in the extractvars task where parenthesis in the variable name will prompt a false warning message.

Add more settings to GEFS replay CI test

cec7b81

Add atmos variables requested for reforeacst

4aaca9a

Reorganize TSNOWP in paramlist

11e1957

Extract ice variables from history file instead

a005e5f

Ice variables are extracted from the history files instead of the product files. This is necessary because there are certain ice variables that are needed that are only available in the history files and not the product files.

Fix typo

2dd9a7e

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Oct 3, 2024

EricSinsky-NOAA reviewed Oct 3, 2024

View reviewed changes

Comment thread parm/config/gefs/config.base Outdated

fix C48_S2SWA_gefs CI error and added HPC_ACCOUNT

922eadf

WalterKolczynski-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Oct 3, 2024

WalterKolczynski-NOAA added the CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules label Oct 7, 2024

emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels Oct 7, 2024

WalterKolczynski-NOAA approved these changes Oct 8, 2024

View reviewed changes

EricSinsky-NOAA mentioned this pull request Oct 15, 2024

Some atmos_ensstat jobs failing for GEFS C384 runs #2856

Closed

EricSinsky-NOAA mentioned this pull request Oct 24, 2024

Bring recent updates from develop and configure reforecast #3015

Merged

38 tasks

Conversation

NeilBarton-NOAA commented Jul 24, 2024

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

NeilBarton-NOAA commented Aug 14, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

emcbot commented Oct 2, 2024

Uh oh!

WalterKolczynski-NOAA commented Oct 2, 2024

Uh oh!

emcbot commented Oct 3, 2024

Uh oh!

Uh oh!

NeilBarton-NOAA commented Oct 3, 2024

Uh oh!

emcbot commented Oct 5, 2024

Uh oh!

emcbot commented Oct 7, 2024

Uh oh!

KateFriedman-NOAA commented Oct 23, 2024

Uh oh!

NeilBarton-NOAA commented Oct 23, 2024

Uh oh!

KateFriedman-NOAA commented Oct 23, 2024

Uh oh!

NeilBarton-NOAA commented Oct 23, 2024

Uh oh!

KateFriedman-NOAA commented Oct 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants