Update MSU glopara paths to new role-global space by KateFriedman-NOAA · Pull Request #3443 · NOAA-EMC/global-workflow

KateFriedman-NOAA · 2025-03-13T18:17:34Z

Description

This PR updates the glopara paths for MSU to use the new role-global space for staged support files.

Resolves #3203

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Workflow hosts file paths for Orion and Hercules

How has this been tested?

Ran Hercules CI manually

Refs NOAA-EMC#3203

…l-msu * origin/develop: Check if HOMEDIR STMP and PTMP are writable (NOAA-EMC#3430) Update UFS_Utils and GFS-utils hashes to update Gaea support and ocean/ice post products (NOAA-EMC#3433) Enable C1152 forecasts on gaea C6 (NOAA-EMC#3438) Migration to role account for Jenkins on Hercules (NOAA-EMC#3423) Remove Direct Linking to COM from DATA for `extractvars` Job (NOAA-EMC#3379)

TerrenceMcGuinness-NOAA · 2025-03-13T21:38:56Z

Looks good. Let's test it with CI on Orion and Hercules simultaneously on the role accounts after PR 3440 goes in.

KateFriedman-NOAA · 2025-03-14T18:05:44Z

@KateFriedman-NOAA @aerorahul Oh wait, no this is different. role.glopara needs to be added to the list of users with HPSS access on C6. Can one of you make this request?

Is this a helpdesk request? If so, which helpdesk? Or somewhere else?

DavidHuber-NOAA · 2025-03-14T18:23:51Z

I think it is an HPSS request (rdhpcs.hpss.help@noaa.gov).

emcbot · 2025-03-14T18:35:16Z

Experiment C48_S2SWA_gefs FAILED on Gaeac6 in Build# 1 with error logs:

/gpfs/f6/drsa-precip3/proj-shared/global/CI/3443/RUNTESTS/COMROOT/C48_S2SWA_gefs_5e585d9f/logs/2021032312/gefs_arch_tars.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-03-14T18:35:24Z

Experiment C48_S2SWA_gefs FAILED on Gaeac6 in Build# 1 in
/gpfs/f6/drsa-precip3/proj-shared/global/CI/3443/RUNTESTS/EXPDIR/C48_S2SWA_gefs_5e585d9f

emcbot · 2025-03-14T18:35:34Z

CI Failed on Gaeac6 in Build# 1
Built and ran in directory /gpfs/f6/drsa-precip3/proj-shared/global/CI/3443


Experiment C48_ATM_5e585d9f Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Fri 14 Mar 2025 01:37:43 PM EDT
Experiment C48_ATM_5e585d9f Terminated: *FAIL*
Error logs:
/gpfs/f6/drsa-precip3/proj-shared/global/CI/3443/RUNTESTS/COMROOT/C48_ATM_5e585d9f/logs/2021032312/gfs_arch_tars.log
Experiment C96C48_hybatmDA_5e585d9f Terminated with 0
FAIL
FAIL tasks failed and 3 dead at Fri 14 Mar 2025 01:42:59 PM EDT
Experiment C96C48_hybatmDA_5e585d9f Terminated: *FAIL*
Error logs:
/gpfs/f6/drsa-precip3/proj-shared/global/CI/3443/RUNTESTS/COMROOT/C96C48_hybatmDA_5e585d9f/logs/2021122100/enkfgdas_earc_tars_00.log
/gpfs/f6/drsa-precip3/proj-shared/global/CI/3443/RUNTESTS/COMROOT/C96C48_hybatmDA_5e585d9f/logs/2021122100/enkfgdas_earc_tars_01.log
/gpfs/f6/drsa-precip3/proj-shared/global/CI/3443/RUNTESTS/COMROOT/C96C48_hybatmDA_5e585d9f/logs/2021122100/gdas_arch_tars.log
Experiment C96_atm3DVar_5e585d9f Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Fri 14 Mar 2025 01:49:37 PM EDT
Experiment C96_atm3DVar_5e585d9f Terminated: *FAIL*
Error logs:
/gpfs/f6/drsa-precip3/proj-shared/global/CI/3443/RUNTESTS/COMROOT/C96_atm3DVar_5e585d9f/logs/2021122100/gdas_arch_tars.log
Experiment C48_S2SW_5e585d9f Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Fri 14 Mar 2025 01:55:33 PM EDT
Experiment C48_S2SW_5e585d9f Terminated: *FAIL*
Error logs:
/gpfs/f6/drsa-precip3/proj-shared/global/CI/3443/RUNTESTS/COMROOT/C48_S2SW_5e585d9f/logs/2021032312/gfs_arch_tars.log
Experiment C48_S2SWA_gefs_5e585d9f Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Fri 14 Mar 2025 02:34:56 PM EDT
Experiment C48_S2SWA_gefs_5e585d9f Terminated: *FAIL*
Error logs:
/gpfs/f6/drsa-precip3/proj-shared/global/CI/3443/RUNTESTS/COMROOT/C48_S2SWA_gefs_5e585d9f/logs/2021032312/gefs_arch_tars.log

KateFriedman-NOAA · 2025-03-14T18:39:10Z

I think it is an HPSS request (rdhpcs.hpss.help@noaa.gov).

Ticket submitted! RDHPCS#2025031454000197

emcbot · 2025-03-14T19:36:38Z

Experiment C48_ATM FAILED on Hercules in Build# 10 with error logs:

/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3443/RUNTESTS/COMROOT/C48_ATM_5e585d9f/logs/2021032312/gfs_atmos_prod_f048-f051.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-03-14T19:36:45Z

Experiment C48_ATM FAILED on Hercules in Build# 10 in
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3443/RUNTESTS/EXPDIR/C48_ATM_5e585d9f

emcbot · 2025-03-14T21:47:33Z

Experiment C48_ATM FAILED on Orion in Build# 12 with error logs:

/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3443/RUNTESTS/COMROOT/C48_ATM_30e91283/logs/2021032312/gfs_atmos_prod_f048-f051.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-03-14T21:47:42Z

Experiment C48_ATM FAILED on Orion in Build# 12 in
/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3443/RUNTESTS/EXPDIR/C48_ATM_30e91283

emcbot · 2025-03-14T21:53:28Z

Experiment C96mx100_S2S FAILED on Orion in Build# 12 with error logs:

/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3443/RUNTESTS/COMROOT/C96mx100_S2S_30e91283/logs/1994050100/sfs_atmos_prod_mem000_f024.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-03-14T21:53:36Z

Experiment C96mx100_S2S FAILED on Orion in Build# 12 in
/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3443/RUNTESTS/EXPDIR/C96mx100_S2S_30e91283

TerrenceMcGuinness-NOAA · 2025-03-14T22:55:06Z

@KateFriedman-NOAA Following the links to error logs there are some grib files missing in the atmos model on both Orion and Hercules.

*** FATAL ERROR: missing input file /work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3443/RUNTESTS/COMROOT/C48_ATM_5e585d9f/gfs.20210323/12//model/atmos/master/gfs.t12z.master.grb2f048 ***

Some tests are passing. I'll list out the passed tests to contrast with these fails next week.

DavidHuber-NOAA · 2025-03-17T11:39:16Z

@TerrenceMcGuinness-NOAA When this error occurs during CI, it usually indicates that one of the tests started over but DATAROOT was not deleted first. This tricks the forecast into thinking that it needs to restart, but the post jobs will fail due to missing data. There are two solutions: 1) delete DATAROOT before starting CI again or 2) update the branch. The latter will work because the CI tests will be from a new hash, meaning the DATAROOT directory will be different. Issue #3071 was opened to address this.

emcbot · 2025-03-17T19:38:58Z

Experiment C96C48_hybatmDA FAILED on Orion in Build# 14 with error logs:

/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3443/RUNTESTS/COMROOT/C96C48_hybatmDA_30e91283/logs/2021122100/gfs_atmos_prod_f117.log
/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3443/RUNTESTS/COMROOT/C96C48_hybatmDA_30e91283/logs/2021122100/gfs_atmos_prod_f120.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-03-17T19:39:09Z

Experiment C96C48_hybatmDA FAILED on Orion in Build# 14 in
/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3443/RUNTESTS/EXPDIR/C96C48_hybatmDA_30e91283

TerrenceMcGuinness-NOAA · 2025-03-17T20:22:19Z

/work2 has become unavailable on both Orion and Hercules. We are working to find the problem and get it resolved as quickly as possible. We will send an update as we learn more.

For any associated problems, please submit a help desk ticket.
HPC2 users email: help@hpc.msstate.edu

Joey B. Jones

emcbot · 2025-03-17T23:07:37Z

CI Failed on Orion in Build# 14
Built and ran in directory /work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3443


Experiment C48_ATM_30e91283 Completed 1 Cycles: *SUCCESS* at Mon Mar 17 01:43:23 PM CDT 2025
Experiment C96mx100_S2S_30e91283 Completed 1 Cycles: *SUCCESS* at Mon Mar 17 02:20:04 PM CDT 2025
Experiment C96C48_hybatmDA_30e91283 Terminated with 0
FAIL
FAIL tasks failed and 2 dead at Mon Mar 17 02:38:38 PM CDT 2025
Experiment C96C48_hybatmDA_30e91283 Terminated: *FAIL*
Error logs:
/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3443/RUNTESTS/COMROOT/C96C48_hybatmDA_30e91283/logs/2021122100/gfs_atmos_prod_f117.log
/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3443/RUNTESTS/COMROOT/C96C48_hybatmDA_30e91283/logs/2021122100/gfs_atmos_prod_f120.log
Experiment C96_atm3DVar_30e91283 Completed 3 Cycles: *SUCCESS* at Mon Mar 17 04:11:37 PM CDT 2025
Experiment C48_S2SWA_gefs_30e91283 Completed 1 Cycles: *SUCCESS* at Mon Mar 17 04:36:11 PM CDT 2025
Experiment C48_S2SW_30e91283 Completed 1 Cycles: *SUCCESS* at Mon Mar 17 06:07:19 PM CDT 2025

emcbot · 2025-03-17T23:54:57Z

CI Passed on Hercules in Build# 13
Built and ran in directory /work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3443


Experiment C48_ATM_30e91283 Completed 1 Cycles: *SUCCESS* at Mon Mar 17 13:21:51 CDT 2025
Experiment C48mx500_hybAOWCDA_30e91283 Completed 2 Cycles: *SUCCESS* at Mon Mar 17 13:52:15 CDT 2025
Experiment C96mx100_S2S_30e91283 Completed 1 Cycles: *SUCCESS* at Mon Mar 17 14:04:13 CDT 2025
Experiment C96_atm3DVar_30e91283 Completed 3 Cycles: *SUCCESS* at Mon Mar 17 15:47:02 CDT 2025
Experiment C96C48_hybatmDA_30e91283 Completed 3 Cycles: *SUCCESS* at Mon Mar 17 15:47:02 CDT 2025
Experiment C48mx500_3DVarAOWCDA_30e91283 Completed 2 Cycles: *SUCCESS* at Mon Mar 17 17:23:48 CDT 2025
Experiment C48_S2SWA_gefs_30e91283 Completed 1 Cycles: *SUCCESS* at Mon Mar 17 18:12:25 CDT 2025
Experiment C48_S2SW_30e91283 Completed 1 Cycles: *SUCCESS* at Mon Mar 17 18:54:38 CDT 2025

emcbot · 2025-03-18T13:07:25Z

CI Passed on Orion in Build# 15
Built and ran in directory /work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3443


Experiment C48_ATM_30e91283 Completed 1 Cycles: *SUCCESS* at Tue Mar 18 05:57:45 AM CDT 2025
Experiment C96mx100_S2S_30e91283 Completed 1 Cycles: *SUCCESS* at Tue Mar 18 06:34:20 AM CDT 2025
Experiment C96C48_hybatmDA_30e91283 Completed 3 Cycles: *SUCCESS* at Tue Mar 18 07:35:51 AM CDT 2025
Experiment C96_atm3DVar_30e91283 Completed 3 Cycles: *SUCCESS* at Tue Mar 18 07:41:56 AM CDT 2025
Experiment C48_S2SW_30e91283 Completed 1 Cycles: *SUCCESS* at Tue Mar 18 07:47:55 AM CDT 2025
Experiment C48_S2SWA_gefs_30e91283 Completed 1 Cycles: *SUCCESS* at Tue Mar 18 08:07:04 AM CDT 2025

DavidHuber-NOAA

Looks good!

KateFriedman-NOAA added 3 commits March 10, 2025 10:58

Update glopara paths to new role-global space

2c93211

Refs NOAA-EMC#3203

Update prepobs and fit2obs paths for role-global

8f8c481

Refs NOAA-EMC#3203

KateFriedman-NOAA requested a review from TerrenceMcGuinness-NOAA March 13, 2025 18:17

KateFriedman-NOAA requested review from DavidHuber-NOAA, WalterKolczynski-NOAA and aerorahul as code owners March 13, 2025 18:17

Merge branch 'NOAA-EMC:develop' into feature/role-global-msu

89c9485

KateFriedman-NOAA removed the request for review from WalterKolczynski-NOAA March 13, 2025 18:26

KateFriedman-NOAA added 2 commits March 14, 2025 08:54

Merge branch 'NOAA-EMC:develop' into feature/role-global-msu

a6d45f6

Merge branch 'develop' into feature/role-global-msu

5e585d9

TerrenceMcGuinness-NOAA added CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion labels Mar 14, 2025

emcbot added the CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion label Mar 14, 2025

Merge branch 'develop' into feature/role-global-msu

edc4ee4

Merge branch 'develop' into feature/role-global-msu

30e9128

Merge branch 'develop' into feature/role-global-msu

f116829

DavidHuber-NOAA approved these changes Mar 18, 2025

View reviewed changes

KateFriedman-NOAA mentioned this pull request Mar 21, 2025

Update MSU FIX_DIR paths #3488

Merged

Conversation

KateFriedman-NOAA commented Mar 13, 2025

Description

Type of change

Change characteristics

How has this been tested?

Uh oh!

TerrenceMcGuinness-NOAA commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KateFriedman-NOAA commented Mar 14, 2025

Uh oh!

DavidHuber-NOAA commented Mar 14, 2025

Uh oh!

emcbot commented Mar 14, 2025

Uh oh!

emcbot commented Mar 14, 2025

Uh oh!

emcbot commented Mar 14, 2025

Uh oh!

KateFriedman-NOAA commented Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

emcbot commented Mar 14, 2025

Uh oh!

emcbot commented Mar 14, 2025

Uh oh!

emcbot commented Mar 14, 2025

Uh oh!

emcbot commented Mar 14, 2025

Uh oh!

emcbot commented Mar 14, 2025

Uh oh!

emcbot commented Mar 14, 2025

Uh oh!

TerrenceMcGuinness-NOAA commented Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DavidHuber-NOAA commented Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

emcbot commented Mar 17, 2025

Uh oh!

emcbot commented Mar 17, 2025

Uh oh!

TerrenceMcGuinness-NOAA commented Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

emcbot commented Mar 17, 2025

Uh oh!

emcbot commented Mar 17, 2025

Uh oh!

emcbot commented Mar 18, 2025

Uh oh!

DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

TerrenceMcGuinness-NOAA commented Mar 13, 2025 •

edited

Loading

KateFriedman-NOAA commented Mar 14, 2025 •

edited

Loading

TerrenceMcGuinness-NOAA commented Mar 14, 2025 •

edited

Loading

DavidHuber-NOAA commented Mar 17, 2025 •

edited

Loading

TerrenceMcGuinness-NOAA commented Mar 17, 2025 •

edited

Loading