Update gaeac5 CI from Anil's PR by TerrenceMcGuinness-NOAA · Pull Request #3419 · NOAA-EMC/global-workflow

TerrenceMcGuinness-NOAA · 2025-03-05T21:26:51Z

Description

This PR is some changes to Node Names for running global-workflow on Gaea C5.
I had to make this a branch from Anil K. PR because he does not have permissions to update the Jenkninsfile on the controller from his PR.

… compatiblity

…te EMC and EPIC naming differences with using dashes

…_nodename Updating MACHINE_ID map to Jenkins Node Name logic for EMC / EPIC compatlity

Updated Node Name for Gaea to GaeaC5 for the "Run with parameters" state.

adding debug lines

AnilKumar-NOAA · 2025-03-06T21:19:55Z

Build FAILED on Gaeac5 in Build# 10 with error logs:
/gpfs/f5/epic/proj-shared/global/CI/3419/global-workflow/sorc/logs/ufs_utils.log
Follow link here to view the contents of the above file(s): (link)

Please take a look on hsi module @DavidHuber-NOAA & @TerrenceMcGuinness-NOAA - Related to hsi update recently in the PR here the hsi update was made in a recent PR of his: https://github.com/NOAA-EMC/global-workflow/pull/3323/files#diff-794f4dde1ed73e1fd0b0c7f4b54b7224da90bfa9225fccae305ea2c5b2ab4352

TerrenceMcGuinness-NOAA · 2025-03-06T21:44:05Z

@DavidHuber-NOAA come to think of David, we do not have custom build options for CI. And yes we just did a merge from develop a couple of hours ago so that may have changed the build.

DavidHuber-NOAA · 2025-03-07T13:14:47Z

@AnilKumar-NOAA Yes, the HSI module is available on C5, but (I think) the DTN partition needs to be updated. The GAEA ES cluster had new nodes added earlier this week and I am unsure of the new partition setup. Are you aware of how it is setup now?

DavidHuber-NOAA · 2025-03-07T13:18:39Z

@AnilKumar-NOAA Apologies, I see now the issue you were referring to. I will look a little deeper.

DavidHuber-NOAA · 2025-03-07T13:34:31Z

The issue is in the ufs_utils modulefiles/build.gaeac5.intel.lua:

help([[
Load environment to compile UFS_UTILS on Gaea using Intel
]])

prepend_path("MODULEPATH", "/sw/rdtn/modulefiles")
load("hsi")

This path needs to be updated to /usw/hpss/modulefiles.

@TerrenceMcGuinness-NOAA You are correct, there is no way to skip this build. Strictly speaking, I don't think this is required by the forecast-only tests, but I don't think we should try to put in a hack. Instead, I will open a UFS_Utils PR now to fix the issue upstream.

DavidBurrows-NCO · 2025-03-07T13:52:40Z

@DavidHuber-NOAA Thanks for looking into this...Let me know if you want EPIC to handle that ufs-utils PR or anything else.

DavidHuber-NOAA · 2025-03-07T13:56:08Z

@DavidBurrows-NCO No problem. It's just a two-line fix. I've already tested it on C5/C6. If George Gayno would like it tested further (CI testing), then I would gladly take assistance.

DavidHuber-NOAA · 2025-03-07T14:01:19Z

Opened PR ufs-community/UFS_UTILS#1031.

DavidBurrows-NCO · 2025-03-07T14:07:10Z

If George Gayno would like it tested further (CI testing), then I would gladly take assistance.

@DavidHuber-NOAA Sounds good. Let us know. We have a full day EPIC meeting today but will check in periodically.

emcbot · 2025-03-07T14:39:19Z

Build FAILED on Gaeac6 in Build# 8 with error logs:

/gpfs/f6/drsa-precip3/proj-shared/global/CI/3419/global-workflow/sorc/logs/ufs_utils.log

Follow link here to view the contents of the above file(s): (link)

TerrenceMcGuinness-NOAA · 2025-03-07T15:22:52Z

Had some issues with Rocoto in role account on Gaea C6. Restarting build to test fix.

emcbot · 2025-03-07T16:17:35Z

Build FAILED on Gaeac6 in Build# 11 with error logs:

/gpfs/f6/drsa-precip3/proj-shared/global/CI/3419/global-workflow/sorc/logs/ufs_utils.log

Follow link here to view the contents of the above file(s): (link)

TerrenceMcGuinness-NOAA · 2025-03-07T16:44:06Z

Just confirmed build fail on develop branch on Gaia C6 with ufs_utils and functional CI test passes with Jenkins updates in this PR with the same fail on this branch (see link above)

AnilKumar-NOAA · 2025-03-07T16:47:58Z

Just confirmed build fail on develop branch on Gaia C6 with ufs_utils and functional CI test passes with Jenkins updates in this PR with the same fail on this branch (see link above)

Yes, hsi issues on c6 as well. Hopefully hsi PR will get merge soon and then we hopefully pass the test atleast on gaeac5.

DavidHuber-NOAA · 2025-03-07T17:19:24Z

@AnilKumar-NOAA @DavidBurrows-NCO Would you like to move forward with this PR as is or wait for a UFS_Utils update? There are a couple of science updates to UFS_Utils between the current hash and what would go in after ufs-community/UFS_UTILS#1031 is merged, which would require comprehensive CI testing on the tier 1 platforms.

DavidBurrows-NCO · 2025-03-07T17:29:12Z

@DavidHuber-NOAA If it's not an issue to move forward, we would prefer that. I'm afraid ufs-utils may take some time to upgrade.

DavidHuber-NOAA

Changes look good.

emcbot · 2025-03-10T14:54:38Z

Build FAILED on Gaeac5 in Build# 12 with error logs:

/gpfs/f5/epic/proj-shared/global/CI/3419/global-workflow/sorc/logs/ufs_utils.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-03-10T16:32:09Z

Build FAILED on Gaeac5 in Build# 14 with error logs:

/gpfs/f5/epic/proj-shared/global/CI/3419__2/global-workflow/sorc/logs/ufs_utils.log

Follow link here to view the contents of the above file(s): (link)

AnilKumar-NOAA and others added 18 commits January 29, 2025 10:49

update the new paths in the config.gaea under ci/platform

a5f2d5a

Rename config.gaea to config.gaeac5

58bf134

Conflict resolved

1e87ca7

Merge branch 'develop' into develop

78b7b1a

added updates to MACHINE_ID to Jenkins Node name map for EMC and EPIC…

35628e5

… compatiblity

updated logic for the MACHINE_ID to Jenkins node name map to accomoda…

96db5cc

…te EMC and EPIC naming differences with using dashes

adding update to single quote on body block that had updated before

7c8b117

turned HPSS arch off for C5

7bc0d1b

removed a no op

811a299

Merge branch 'NOAA-EMC:develop' into develop

a48e26d

Merge branch 'NOAA-EMC:develop' into develop

759aa71

Merge branch 'NOAA-EMC:develop' into develop

b83795a

Updated gaea to gaeac5 in ci dir

e3771d4

Merge branch 'update_gaeac5_ci' into hotfix_jenkins_emc_nodename

64c3a56

Merge pull request #9 from TerrenceMcGuinness-NOAA/hotfix_jenkins_emc…

539e67c

…_nodename Updating MACHINE_ID map to Jenkins Node Name logic for EMC / EPIC compatlity

Merge branch 'NOAA-EMC:develop' into update_gaeac5_ci

e78f0da

Update Jenkinsfile

ac5c914

Updated Node Name for Gaea to GaeaC5 for the "Run with parameters" state.

Update Jenkinsfile

f5fbf15

adding debug lines

TerrenceMcGuinness-NOAA requested a review from WalterKolczynski-NOAA as a code owner March 5, 2025 21:26

TerrenceMcGuinness-NOAA changed the title ~~Update gaeac5 ci anil~~ Update gaeac5 CI from Anil's PR Mar 5, 2025

TerrenceMcGuinness-NOAA added CI/CD Issue related to CI/CD CI-Gaeac5-Ready labels Mar 5, 2025

emcbot added CI-Gaeac5-Building and removed CI-Gaeac5-Ready labels Mar 5, 2025

chainging account to epic on Gaea C5

b74dec6

TerrenceMcGuinness-NOAA requested review from DavidHuber-NOAA and aerorahul as code owners March 5, 2025 22:20

TerrenceMcGuinness-NOAA removed the CI-Gaeac5-Failed label Mar 5, 2025

emcbot added CI-Gaeac6-Failed **Bot use only** CI testing on Gaea C6 for this PR has failed and removed CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 labels Mar 7, 2025

TerrenceMcGuinness-NOAA added CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 and removed CI-Gaeac6-Failed **Bot use only** CI testing on Gaea C6 for this PR has failed labels Mar 7, 2025

emcbot added CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 and removed CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 labels Mar 7, 2025

emcbot added CI-Gaeac6-Failed **Bot use only** CI testing on Gaea C6 for this PR has failed and removed CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 labels Mar 7, 2025

DavidHuber-NOAA previously approved these changes Mar 7, 2025

View reviewed changes

TerrenceMcGuinness-NOAA added CI-Gaeac5-Ready and removed CI-Gaeac5-Failed labels Mar 10, 2025

Merge branch 'develop' into update_gaeac5_ci_anil

d96082d

Conversation

TerrenceMcGuinness-NOAA commented Mar 5, 2025

Description

Uh oh!

AnilKumar-NOAA commented Mar 6, 2025

Uh oh!

TerrenceMcGuinness-NOAA commented Mar 6, 2025

Uh oh!

DavidHuber-NOAA commented Mar 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DavidHuber-NOAA commented Mar 7, 2025

Uh oh!

DavidHuber-NOAA commented Mar 7, 2025

Uh oh!

DavidBurrows-NCO commented Mar 7, 2025

Uh oh!

DavidHuber-NOAA commented Mar 7, 2025

Uh oh!

DavidHuber-NOAA commented Mar 7, 2025

Uh oh!

DavidBurrows-NCO commented Mar 7, 2025

Uh oh!

emcbot commented Mar 7, 2025

Uh oh!

TerrenceMcGuinness-NOAA commented Mar 7, 2025

Uh oh!

emcbot commented Mar 7, 2025

Uh oh!

TerrenceMcGuinness-NOAA commented Mar 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AnilKumar-NOAA commented Mar 7, 2025

Uh oh!

DavidHuber-NOAA commented Mar 7, 2025

Uh oh!

DavidBurrows-NCO commented Mar 7, 2025

Uh oh!

DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

emcbot commented Mar 10, 2025

Uh oh!

emcbot commented Mar 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

DavidHuber-NOAA commented Mar 7, 2025 •

edited

Loading

TerrenceMcGuinness-NOAA commented Mar 7, 2025 •

edited

Loading