Switch to Rocky 9 built external packages on Hercules by WalterKolczynski-NOAA · Pull Request #2608 · NOAA-EMC/global-workflow

WalterKolczynski-NOAA · 2024-05-20T05:08:02Z

Description

The workflow was updated to use modules built on Rocky 9, but the external packages (like prepobs) were still pointing to the versions built on CentOS (Orion). This transitions to packages built on Rocky 9.

Updating of the tracker package has been deferred until later. As such, the tracker jobs have been disabled by returning immediately if they are on Hercules. Since these jobs are small, resource-wise, it should not meaningfully impact turnover time.

Type of change

New feature (adds functionality)

Change characteristics

Is this a breaking change (a change in existing functionality)? YES
- Tracker has not yet been updated to build on Hercules
Does this change require a documentation update? NO

How has this been tested?

Cycled test on Hercules

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
I have made corresponding changes to the documentation if necessary

WalterKolczynski-NOAA · 2024-05-20T05:16:29Z

The only update needed to the packages was a minor change to the Fit2Obs spack-stack module path:

-prepend_path("MODULEPATH", "/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/unified-env/install/modulefiles/Core")
+prepend_path("MODULEPATH", "/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/gsi-addon-env/install/modulefiles/Core")

Changes are made in-place without changing the version number of the directory (yet).

emcbot · 2024-05-20T06:43:33Z

Experiment C48_ATM FAILED on Hercules with error logs:

/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/COMROOT/C48_ATM_1f1ac3e4/logs/2021032312/gfsgenesis.log
/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/COMROOT/C48_ATM_1f1ac3e4/logs/2021032312/gfstracker.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-05-20T06:43:37Z

Experiment C48_S2SW FAILED on Hercules in
/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/C48_S2SW_1f1ac3e4

emcbot · 2024-05-20T06:43:43Z

Experiment C48_ATM FAILED on Hercules in
/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/C48_ATM_1f1ac3e4

emcbot · 2024-05-20T07:35:26Z

Experiment C96_atm3DVar FAILED on Hercules with error logs:

/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/COMROOT/C96_atm3DVar_1f1ac3e4/logs/2021122100/gfsgenesis.log
/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/COMROOT/C96_atm3DVar_1f1ac3e4/logs/2021122100/gfstracker.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-05-20T07:35:31Z

Experiment C96C48_hybatmDA FAILED on Hercules in
/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/C96C48_hybatmDA_1f1ac3e4

emcbot · 2024-05-20T07:35:35Z

Experiment C96_atm3DVar FAILED on Hercules in
/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/C96_atm3DVar_1f1ac3e4

DavidHuber-NOAA · 2024-05-20T11:50:38Z

It looks like the TC_Tracker is not in the specified location:

hercules-login-1[3] ~$ ls /work/noaa/global/glopara/git_rocky9/TC_tracker/
hercules-login-1[4] ~$

WalterKolczynski-NOAA · 2024-05-20T16:31:39Z

It looks like the TC_Tracker is not in the specified location:
hercules-login-1[3] ~$ ls /work/noaa/global/glopara/git_rocky9/TC_tracker/
hercules-login-1[4] ~$

Yeah, this was a known thing. I thought I had tested this before I launched CI, but might've mixed it up with another test I did over the weekend.

The workflow was updated to use modules built on Rocky 9, but the external packages (like prepobs) were still pointing to the versions built on CentOS (Orion). This transitions to packages built on Rocky 9.

The external tracker package is not yet available on Rocky 9, so a hack is added to skip those jobs on Hercules until the package is available. This will need to be extended to Orion if that machine transitions to Rocky 9 before a package is available. Refs NOAA-EMC#2639

emcbot · 2024-05-30T14:30:31Z

Automated global-workflow Testing Results:

Machine: Wcoss2
Start: Thu May 30 14:17:49 UTC 2024 on clogin01
---------------------------------------------------
Build: Completed at 05/30/24 02:30:19 PM
Case setup: Completed for experiment C48_ATM_d45b7ca5
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_d45b7ca5
Case setup: Skipped for experiment C48_S2SWA_gefs_d45b7ca5
Case setup: Completed for experiment C48_S2SW_d45b7ca5
Case setup: Completed for experiment C96_atm3DVar_extended_d45b7ca5
Case setup: Skipped for experiment C96_atm3DVar_d45b7ca5
Case setup: Skipped for experiment C96_atmaerosnowDA_d45b7ca5
Case setup: Completed for experiment C96C48_hybatmDA_d45b7ca5
Case setup: Skipped for experiment C96C48_ufs_hybatmDA_d45b7ca5

emcbot · 2024-05-30T17:48:31Z

Experiment C48_S2SW FAILED on Hera with error logs:

/scratch1/NCEPDEV/global/CI/2608/RUNTESTS/COMROOT/C48_S2SW_d45b7ca5/logs/2021032312/gfswaveinit.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-05-30T17:48:32Z

Experiment C96_atm3DVar FAILED on Hera in
/scratch1/NCEPDEV/global/CI/2608/RUNTESTS/C96_atm3DVar_d45b7ca5

emcbot · 2024-05-30T17:48:32Z

Experiment C96_atmaerosnowDA FAILED on Hera in
/scratch1/NCEPDEV/global/CI/2608/RUNTESTS/C96_atmaerosnowDA_d45b7ca5

emcbot · 2024-05-30T17:48:32Z

Experiment C48mx500_3DVarAOWCDA FAILED on Hera in
/scratch1/NCEPDEV/global/CI/2608/RUNTESTS/C48mx500_3DVarAOWCDA_d45b7ca5

emcbot · 2024-05-30T17:48:33Z

Experiment C96C48_hybatmDA FAILED on Hera in
/scratch1/NCEPDEV/global/CI/2608/RUNTESTS/C96C48_hybatmDA_d45b7ca5

emcbot · 2024-05-30T17:48:39Z

Experiment C48_S2SW FAILED on Hera in
/scratch1/NCEPDEV/global/CI/2608/RUNTESTS/C48_S2SW_d45b7ca5

emcbot · 2024-05-30T17:54:33Z

Experiment C48_ATM FAILED on Hera with error logs:

/scratch1/NCEPDEV/global/CI/2608/RUNTESTS/COMROOT/C48_ATM_d45b7ca5/logs/2021032312/gfsfcst.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-05-30T17:54:33Z

Experiment C48_S2SWA_gefs FAILED on Hera in
/scratch1/NCEPDEV/global/CI/2608/RUNTESTS/C48_S2SWA_gefs_d45b7ca5

emcbot · 2024-05-30T17:54:38Z

Experiment C48_ATM FAILED on Hera in
/scratch1/NCEPDEV/global/CI/2608/RUNTESTS/C48_ATM_d45b7ca5

emcbot · 2024-05-30T18:21:56Z

CI Passed Orion at
Built and ran in directory /work2/noaa/stmp/CI/ORION/2608

emcbot · 2024-05-30T18:27:16Z

Experiment C48_ATM_d45b7ca5 SUCCESS on Wcoss2 at 05/30/24 06:27:15 PM

emcbot · 2024-05-30T18:27:20Z

Experiment C48_S2SW_d45b7ca5 SUCCESS on Wcoss2 at 05/30/24 06:27:19 PM

emcbot · 2024-05-30T18:42:23Z

Experiment C96C48_hybatmDA_d45b7ca5 SUCCESS on Wcoss2 at 05/30/24 06:42:22 PM

emcbot · 2024-05-31T06:33:26Z

Experiment C96_atm3DVar_extended_d45b7ca5 SUCCESS on Wcoss2 at 05/31/24 06:33:26 AM

emcbot · 2024-05-31T06:36:26Z

All CI Test Cases Passed on Wcoss2:

Experiment C48_ATM_d45b7ca5 *** SUCCESS *** at 05/30/24 06:27:15 PM
Experiment C48_S2SW_d45b7ca5 *** SUCCESS *** at 05/30/24 06:27:19 PM
Experiment C96C48_hybatmDA_d45b7ca5 *** SUCCESS *** at 05/30/24 06:42:22 PM
Experiment C96_atm3DVar_extended_d45b7ca5 *** SUCCESS *** at 05/31/24 06:33:26 AM

emcbot · 2024-05-31T07:10:50Z

CI Passed Hera at
Built and ran in directory /scratch1/NCEPDEV/global/CI/2608

WalterKolczynski-NOAA requested review from DavidHuber-NOAA, KateFriedman-NOAA and aerorahul May 20, 2024 05:08

WalterKolczynski-NOAA self-assigned this May 20, 2024

WalterKolczynski-NOAA added the CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules label May 20, 2024

WalterKolczynski-NOAA force-pushed the feature/migrate_external_rocky9 branch from 85d8dba to 1f1ac3e Compare May 20, 2024 05:10

emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules labels May 20, 2024

emcbot added CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress and removed CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels May 20, 2024

emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels May 20, 2024

WalterKolczynski-NOAA removed the CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed label May 29, 2024

WalterKolczynski-NOAA mentioned this pull request May 30, 2024

Re-enable tracker jobs on Hercules and Orion #2639

Closed

1 task

Switch to Rocky 9 built external packages on Hercules

2a7211e

The workflow was updated to use modules built on Rocky 9, but the external packages (like prepobs) were still pointing to the versions built on CentOS (Orion). This transitions to packages built on Rocky 9.

WalterKolczynski-NOAA force-pushed the feature/migrate_external_rocky9 branch from 1f1ac3e to 8904861 Compare May 30, 2024 01:40

WalterKolczynski-NOAA force-pushed the feature/migrate_external_rocky9 branch from 8904861 to ecac748 Compare May 30, 2024 01:42

WalterKolczynski-NOAA added the CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules label May 30, 2024

aerorahul approved these changes May 30, 2024

View reviewed changes

emcbot removed the CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules label May 30, 2024

emcbot added CI-Wcoss2-Running CI testing on WCOSS for this PR is in-progress and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Wcoss2-Building CI testing is cloning/building on WCOSS2 labels May 30, 2024

TerrenceMcGuinness-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels May 30, 2024

Conversation

WalterKolczynski-NOAA commented May 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

WalterKolczynski-NOAA commented May 20, 2024

Uh oh!

emcbot commented May 20, 2024

Uh oh!

emcbot commented May 20, 2024

Uh oh!

emcbot commented May 20, 2024

Uh oh!

emcbot commented May 20, 2024

Uh oh!

emcbot commented May 20, 2024

Uh oh!

emcbot commented May 20, 2024

Uh oh!

DavidHuber-NOAA commented May 20, 2024

Uh oh!

WalterKolczynski-NOAA commented May 20, 2024

Uh oh!

emcbot commented May 30, 2024

Uh oh!

emcbot commented May 30, 2024

Uh oh!

emcbot commented May 30, 2024

Uh oh!

emcbot commented May 30, 2024

Uh oh!

emcbot commented May 30, 2024

Uh oh!

emcbot commented May 30, 2024

Uh oh!

emcbot commented May 30, 2024

Uh oh!

emcbot commented May 30, 2024

Uh oh!

emcbot commented May 30, 2024

Uh oh!

emcbot commented May 30, 2024

Uh oh!

emcbot commented May 30, 2024

Uh oh!

emcbot commented May 30, 2024

Uh oh!

emcbot commented May 30, 2024

Uh oh!

emcbot commented May 30, 2024

Uh oh!

emcbot commented May 31, 2024

Uh oh!

emcbot commented May 31, 2024

Uh oh!

emcbot commented May 31, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

WalterKolczynski-NOAA commented May 20, 2024 •

edited

Loading