Add support for forecast-only runs on AWS by weihuang-jedi · Pull Request #2711 · NOAA-EMC/global-workflow

weihuang-jedi · 2024-06-22T03:28:52Z

Description

The purpose of this PR it to merge code that allowing global-workflow run on AWS, first focus on ATM forecast only.

Resolves: #2709

Type of change

New feature (adds functionality)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? YES

How has this been tested?

Clone and build on AWS
ATM Forecast-only on AWS

Checklist

[x ] Any dependent changes have been merged and published
[x ] My code follows the style guidelines of this project
[x ] I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
[ x] My changes generate no new warnings
New and existing tests pass with my changes
I have made corresponding changes to the documentation if necessary

…d hope can provide a better way

…-workflow-cloud into aws-forecast-only

…d hope can provide a better way

aerorahul

some comments regarding mpmd_opt and wall time for atmos_products.

weihuang-jedi · 2024-07-01T14:39:20Z

@aerorahul I have replied your comments, and make most of the changes you requested.
There are few left:

remove NCL, since which cause compile error, and others have NCL as well, can we leave it as is?
git version, that goes with spack-stack, can we just follow spack-stack?
the "mpmd_opt" did not make negative impacts so far for forecast, do you want remove it from forecast and only keep in atmos-products?
in tasks.py, and workflow_xml.py we use "pw_csp" local variable, some used twice and more. do you prefer not use local variable, but use "os.get_environ_get" directly.

Thanks,

Wei

emcbot · 2024-08-06T20:57:52Z

Experiment C96_atm3DVar_extended_06093aff FAIL on Wcoss2 at 08/06/24 08:57:51 PM

Error logs:

/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2711/RUNTESTS/COMROOT/C96_atm3DVar_extended_06093aff/logs/2021122100/gfsgempakgrb2spec_f000.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2711/RUNTESTS/COMROOT/C96_atm3DVar_extended_06093aff/logs/2021122100/gfsgempakgrb2spec_f003.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2711/RUNTESTS/COMROOT/C96_atm3DVar_extended_06093aff/logs/2021122100/gfsgempakgrb2spec_f006.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2711/RUNTESTS/COMROOT/C96_atm3DVar_extended_06093aff/logs/2021122100/gfsgempakgrb2spec_f009.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2711/RUNTESTS/COMROOT/C96_atm3DVar_extended_06093aff/logs/2021122100/gfsgempakgrb2spec_f012.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2711/RUNTESTS/COMROOT/C96_atm3DVar_extended_06093aff/logs/2021122100/gfsgempakgrb2spec_f015.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2711/RUNTESTS/COMROOT/C96_atm3DVar_extended_06093aff/logs/2021122100/gfsgempakgrb2spec_f018.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2711/RUNTESTS/COMROOT/C96_atm3DVar_extended_06093aff/logs/2021122100/gfsgempakgrb2spec_f021.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2711/RUNTESTS/COMROOT/C96_atm3DVar_extended_06093aff/logs/2021122100/gfsgempakgrb2spec_f024.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2711/RUNTESTS/COMROOT/C96_atm3DVar_extended_06093aff/logs/2021122100/gfsgempakgrb2spec_f027.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2711/RUNTESTS/COMROOT/C96_atm3DVar_extended_06093aff/logs/2021122100/gfsgempakgrb2spec_f030.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2711/RUNTESTS/COMROOT/C96_atm3DVar_extended_06093aff/logs/2021122100/gfsgempakgrb2spec_f033.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2711/RUNTESTS/COMROOT/C96_atm3DVar_extended_06093aff/logs/2021122100/gfsgempakgrb2spec_f036.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2711/RUNTESTS/COMROOT/C96_atm3DVar_extended_06093aff/logs/2021122100/gfsgempakgrb2spec_f039.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2711/RUNTESTS/COMROOT/C96_atm3DVar_extended_06093aff/logs/2021122100/gfsgempakgrb2spec_f042.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2711/RUNTESTS/COMROOT/C96_atm3DVar_extended_06093aff/logs/2021122100/gfsgempakgrb2spec_f045.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2711/RUNTESTS/COMROOT/C96_atm3DVar_extended_06093aff/logs/2021122100/gfsgempakgrb2spec_f048.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2711/RUNTESTS/COMROOT/C96_atm3DVar_extended_06093aff/logs/2021122100/gfsgempakgrb2spec_f051.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2711/RUNTESTS/COMROOT/C96_atm3DVar_extended_06093aff/logs/2021122100/gfsgempakgrb2spec_f054.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2711/RUNTESTS/COMROOT/C96_atm3DVar_extended_06093aff/logs/2021122100/gfsgempakgrb2spec_f057.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2711/RUNTESTS/COMROOT/C96_atm3DVar_extended_06093aff/logs/2021122100/gfsgempakgrb2spec_f060.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2711/RUNTESTS/COMROOT/C96_atm3DVar_extended_06093aff/logs/2021122100/gfsgempakmeta.log

Follow link here to view the contents of the above file(s): (link)

weihuang-jedi · 2024-08-08T14:00:26Z

@WalterKolczynski-NOAA Is there anything more that I need to do to get this PR merged?

emcbot · 2024-08-12T17:30:59Z

CI Update on Wcoss2 at 08/12/24 05:30:58 PM
============================================
Cloning and Building global-workflow PR: 2711
with PID: 219881 on host: clogin03

emcbot · 2024-08-12T18:41:01Z

Automated global-workflow Testing Results:

Machine: Wcoss2
Start: Mon Aug 12 17:36:25 UTC 2024 on clogin03
---------------------------------------------------
Build: Completed at 08/12/24 06:40:40 PM
Case setup: Completed for experiment C48_ATM_bd1c9549
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_bd1c9549
Case setup: Skipped for experiment C48_S2SWA_gefs_bd1c9549
Case setup: Completed for experiment C48_S2SW_bd1c9549
Case setup: Completed for experiment C96_atm3DVar_extended_bd1c9549
Case setup: Skipped for experiment C96_atm3DVar_bd1c9549
Case setup: Completed for experiment C96_atmaerosnowDA_bd1c9549
Case setup: Completed for experiment C96C48_hybatmDA_bd1c9549
Case setup: Completed for experiment C96C48_ufs_hybatmDA_bd1c9549

emcbot · 2024-08-13T05:28:13Z

All CI Test Cases Passed on Wcoss2:

Experiment C48_ATM_bd1c9549 *** SUCCESS *** at 08/12/24 07:49:14 PM
Experiment C48_S2SW_bd1c9549 *** SUCCESS *** at 08/12/24 07:56:13 PM
Experiment C96C48_hybatmDA_bd1c9549 *** SUCCESS *** at 08/12/24 08:42:24 PM
Experiment C96_atmaerosnowDA_bd1c9549 *** SUCCESS *** at 08/12/24 09:35:24 PM
Experiment C96C48_ufs_hybatmDA_bd1c9549 *** SUCCESS *** at 08/12/24 09:56:17 PM
Experiment C96_atm3DVar_extended_bd1c9549 *** SUCCESS *** at 08/13/24 05:21:32 AM

Addressed

…e_rocoto * origin/develop: Jenkins Pipeline Updates (NOAA-EMC#2815) Add Gaea C5 to CI (NOAA-EMC#2814) Add support for forecast-only runs on AWS (NOAA-EMC#2711) Add fixes to products for when REPLAY IC's are used (NOAA-EMC#2755) Add capability to run forecast in segments (NOAA-EMC#2795)

Changes to make GEFS C48 case run on AWS. After C48 ATM forecast only runs on AWs, the next step is to make GEFS C48 run on AWS. Changes to AWS env, and yaml files. Resolves #2817 Refs #2711

Support global-workflow ATM forecast only runs on Google. Add/Modify env, yaml, and python scripts changes to make global-workflow ATM forecast only runs on GSP. Resolves #2831 Refs #2826 Refs #2711

weihuang-jedi and others added 8 commits June 18, 2024 23:05

compiled OK now

103f2c4

Merge branch 'NOAA-EMC:develop' into aws-forecast-only

916ff6c

re-test on aws with fewer changes

b0ac406

make change in tasks.py to avoid error finding libiomp5.so problem

3de972f

Merge branch 'NOAA-EMC:develop' into aws-forecast-only

8308375

add comments so the reviewers know that these changes are for AWS, an…

bc4c4a8

…d hope can provide a better way

Merge branch 'aws-forecast-only' of ssh://github.com/NOAA-EPIC/global…

924aede

…-workflow-cloud into aws-forecast-only

add comments so the reviewers know that these changes are for AWS, an…

b724937

…d hope can provide a better way

weihuang-jedi marked this pull request as ready for review June 23, 2024 18:47

github-advanced-security AI found potential problems Jun 24, 2024

View reviewed changes

Comment thread ush/forecast_postdet.sh Fixed

aerorahul requested changes Jun 24, 2024

View reviewed changes

DavidHuber-NOAA mentioned this pull request Jun 25, 2024

Implement global-workflow on AWS #2549

Closed

weihuang-jedi and others added 5 commits June 25, 2024 20:56

reverse config.resource changes, and memory restriction on AWS

12ab29f

sync with emc repo

adff250

move common data to a shared place

2290ea2

use ICs from s3-bucket

cd2c8e7

Merge branch 'develop' into aws-forecast-only

4e144e5

WalterKolczynski-NOAA changed the title ~~Make global-workflow run on AWS -- focus on ATM forecast only first~~ Add support for forecast-only runs on AWS Jun 27, 2024

weihuang-jedi added 5 commits July 2, 2024 13:44

change as suggested by reviewer

46e3ef5

sync with develop

32f13eb

sync sorc/ufs_model.fd

a34a4c8

remove mpmd_opt from APRUN_UFS

44011a3

mpmd_opt and switch off tracker/genesis default for AWS

965ec80

aerorahul reviewed Jul 5, 2024

View reviewed changes

Comment thread parm/config/gfs/config.base

add TODO

3ce268e

DavidHuber-NOAA reviewed Jul 5, 2024

View reviewed changes

Comment thread modulefiles/module_base.noaacloud.lua Outdated

weihuang-jedi added 3 commits July 6, 2024 22:02

remove ncl version on AWS

f03ac78

Merge remote-tracking branch 'origin/develop' into aws-forecast-only

007a56b

sync ufs_model

2f6ec6e

Merge branch 'NOAA-EMC:develop' into aws-forecast-only

6fff724

emcbot added CI-Wcoss2-Failed CI testing on WCOSS for this PR has failed and removed CI-Wcoss2-Running CI testing on WCOSS for this PR is in-progress labels Aug 6, 2024

Merge branch 'NOAA-EMC:develop' into aws-forecast-only

f23d2d0

WalterKolczynski-NOAA removed the CI-Wcoss2-Failed CI testing on WCOSS for this PR has failed label Aug 8, 2024

Merge branch 'NOAA-EMC:develop' into aws-forecast-only

bd1c954

weihuang-jedi mentioned this pull request Aug 8, 2024

Add GEFS C48 support on AWS #2818

Merged

12 tasks

weihuang-jedi requested a review from DavidHuber-NOAA August 9, 2024 21:20

WalterKolczynski-NOAA added the CI-Wcoss2-Ready PR is ready for CI testing on WCOSS2. label Aug 12, 2024

emcbot added CI-Wcoss2-Building CI testing is cloning/building on WCOSS2 and removed CI-Wcoss2-Ready PR is ready for CI testing on WCOSS2. labels Aug 12, 2024

emcbot added CI-Wcoss2-Running CI testing on WCOSS for this PR is in-progress and removed CI-Wcoss2-Building CI testing is cloning/building on WCOSS2 labels Aug 12, 2024

Merge branch 'NOAA-EMC:develop' into aws-forecast-only

d12d9e9

emcbot added CI-Wcoss2-Passed CI testing on WCOSS for this PR has completed successfully and removed CI-Wcoss2-Running CI testing on WCOSS for this PR is in-progress labels Aug 13, 2024

WalterKolczynski-NOAA merged commit eba813f into NOAA-EMC:develop Aug 13, 2024

This was referenced Aug 13, 2024

Implementing global-workflow on AWS #2583

Closed

support ATM forecast only on Azure #2827

Merged

Support ATM forecast only on Google #2832

Merged

weihuang-jedi deleted the aws-forecast-only branch August 22, 2024 13:35

Conversation

weihuang-jedi commented Jun 22, 2024

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

Uh oh!

aerorahul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

weihuang-jedi commented Jul 1, 2024

Uh oh!

Uh oh!

Uh oh!

emcbot commented Aug 6, 2024

Uh oh!

weihuang-jedi commented Aug 8, 2024

Uh oh!

emcbot commented Aug 12, 2024

Uh oh!

emcbot commented Aug 12, 2024

Uh oh!

emcbot commented Aug 13, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants