Upgrade to spack-stack 1.9.2 by DavidHuber-NOAA · Pull Request #3798 · NOAA-EMC/global-workflow

DavidHuber-NOAA · 2025-06-13T15:10:37Z

Description

This updates the global workflow to spack-stack 1.9.2 as well as the subcomponents. This PR also includes a partial port to Ursa, though no global workflow tests have been nor can be run there yet.

Fixes #2984
Fixes #3920
Fixes #3921
Fixes #3922
Fixes #3923
Fixes #3852
Fixes #2756
Fixes #3934

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? YES
Does this change require an update to any of the following submodules? YES
- GDAS Update GDASApp to spack-stack/1.9.2 GDASApp#1684
- GFS-utils Upgrade to spack-stack 1.9.2, update WCOSS2 modules gfs-utils#114
- GSI Upgrade to spack-stack 1.9.2 on all RDHPCS platforms GSI#890
- GSI-monitor Add support for Ursa, upgrade to spack-stack 1.9.1 GSI-Monitor#185
- GSI-utils Upgrade to spack-stack 1.9.2 GSI-utils#78
- UFS-utils Upgrade to spack-stack 1.9.2 ufs-community/UFS_UTILS#1076
- UFS-weather-model Update GDASApp to spack-stack/1.9.2 GDASApp#1684

How has this been tested?

Full suite of tests on all platforms

Checklist

…low into feature/191

… to 1.9.1

DavidHuber-NOAA · 2025-08-15T14:20:52Z

@JessicaMeixner-NOAA @bbakernoaa @aerorahul The C96mx100_S2S test is failing during the forecast step due to apparent instabilities on Hercules:

FATAL from PE     1: There were a total of     43254 locations detected with extreme surface values!

The log file can be found here: /work2/noaa/stmp/dhuber/HERCULES/para_192d/COMROOT/C96mx100_S2S_192d/logs/1994050100/sfs_fcst_mem000_seg0.log

I am inclined to disable this test on Hercules and open an issue to investigate it. Thoughts?

JessicaMeixner-NOAA · 2025-08-15T14:22:44Z

I agree - however, SFS is using hercules for a lot of their runs, so we should also have @NeilBarton-NOAA and @XiaqiongZhou-NOAA weigh in.

NeilBarton-NOAA · 2025-08-15T14:25:31Z

This test should not be deleted. I'll look at the logs

DavidHuber-NOAA · 2025-08-15T14:27:57Z

@NeilBarton-NOAA Just clarifying that I'm not proposing that we delete the test. I would just make the following change:

--- a/dev/ci/cases/pr/C96mx100_S2S.yaml
+++ b/dev/ci/cases/pr/C96mx100_S2S.yaml
@@ -20,3 +20,4 @@ arguments:
 skip_ci_on_hosts:
   - gaeac5
   - awsepicglobalworkflow
+  - hercules

This would disable the test on Hercules while the issue can be investigated.

NeilBarton-NOAA · 2025-08-15T14:30:28Z

SFS routinely runs on hercules.

DavidHuber-NOAA · 2025-08-15T14:36:02Z

I wonder if this could be a related issue to the change to the stack-oneapi module file seen on 8/13 (noted here). Could there be an issue with the SFS model building with Intel LLVM compilers?

JessicaMeixner-NOAA · 2025-08-15T14:38:32Z

I have successful runs of S2S from 8/14: /work2/noaa/marine/jmeixner/hercules/gw3798_20250814/t3798/COMROOT/C48_S2SW_t3798

and the C96:

/work2/noaa/marine/jmeixner/hercules/gw3798_20250814/t3798/COMROOT/C96mx100_S2S_t3798/logs

DavidHuber-NOAA · 2025-08-15T14:43:03Z

@JessicaMeixner-NOAA Could you run these to open up permissions on the log directories?

find /work2/noaa/marine/jmeixner/hercules/gw3798_20250814/t3798/COMROOT/* -type d -exec chmod o+rx {} +
find /work2/noaa/marine/jmeixner/hercules/gw3798_20250814/t3798/COMROOT/* -type f -exec chmod o+r {} +

CatherineThomas-NOAA · 2025-08-15T14:43:07Z

Tagging @EdwardSafford-NOAA for awareness:

Ursa update:

We need a minor change in ursa.env - I'll post in code with that update.

vminmod jobs are failing

sample logs: /scratch3/NCEPDEV/climate/Jessica.Meixner/gwpr3798/ut03/COMROOT/C48mx500_3DVarAOWCDA_ut03/logs/2021032500/gfs_vminmon.log.0

@JessicaMeixner-NOAA This indicates that an additional Perl module is required. I made the same request for other systems after the Rocky 8 updates. I'll make a request shortly to the RDHPCS help desk.

JessicaMeixner-NOAA · 2025-08-15T14:45:14Z

@DavidNew-NOAA @RussTreadon-NOAA

I'm running this branch on Ursa, and for C96C48_ufs_hybatmDA atmanlvar and atmensanlobs both run out of memory. See logs here: /scratch3/NCEPDEV/climate/Jessica.Meixner/gwpr3798/ut03/COMROOT/C96C48_ufs_hybatmDA_ut03/logs/2024022400/

JessicaMeixner-NOAA · 2025-08-15T14:50:25Z

@CoryMartin-NOAA - An out of memory error on Ursa: /scratch3/NCEPDEV/climate/Jessica.Meixner/gwpr3798/ut03/COMROOT/C96_gcafs_cycled_ut03/logs/2021122018/gcdas_aeroanlvar.log.0

@bbakernoaa @lipan-NOAA - I don't quite understand this error, b/c I don't think these jobs actually ran out of walltime, but the gcafs forecasts are failling w/"out of walltime" on Ursa using this branch:
Logs are here:
/scratch3/NCEPDEV/climate/Jessica.Meixner/gwpr3798/ut03/COMROOT/C96_gcafs_cycled_ut03/logs
/scratch3/NCEPDEV/climate/Jessica.Meixner/gwpr3798/ut03/COMROOT/C96_gcafs_cycled_noDA_ut03/logs

CoryMartin-NOAA · 2025-08-15T14:59:55Z

Are we only using one node on Ursa? I see 96 cores. Can we change it to max 48 tasks per node for Ursa for the aeroanl job?

bbakernoaa · 2025-08-15T15:02:15Z

@CoryMartin-NOAA - An out of memory error on Ursa: /scratch3/NCEPDEV/climate/Jessica.Meixner/gwpr3798/ut03/COMROOT/C96_gcafs_cycled_ut03/logs/2021122018/gcdas_aeroanlvar.log.0

@bbakernoaa @lipan-NOAA - I don't quite understand this error, b/c I don't think these jobs actually ran out of walltime, but the gcafs forecasts are failling w/"out of walltime" on Ursa using this branch: Logs are here: /scratch3/NCEPDEV/climate/Jessica.Meixner/gwpr3798/ut03/COMROOT/C96_gcafs_cycled_ut03/logs /scratch3/NCEPDEV/climate/Jessica.Meixner/gwpr3798/ut03/COMROOT/C96_gcafs_cycled_noDA_ut03/logs

/scratch3/NCEPDEV/climate/Jessica.Meixner/gwpr3798/ut03/COMROOT/C96_gcafs_cycled_ut03/logs/2021122012/gcdas_fcst_seg0.log looks like it completed. Am I missing something?

DavidHuber-NOAA · 2025-08-15T16:07:31Z

I've been informed that the spack-stack issues have been resolved on Hercules and Orion. I'm going to recompile the SFS model and relaunch the forecasts to see if that resolves the issue.

RussTreadon-NOAA · 2025-08-15T16:23:04Z

Confirmed that GSI builds on Orion and Hercules now load the correct fortran compiler.

-- The C compiler identification is IntelLLVM 2024.2.1
-- The Fortran compiler identification is Intel 2021.1.0.20240703

Test done with GSI develop at 3be73271. This is the commit prior to adding

setenv("CC","mpiicc")
setenv("CXX","mpiicpc")
setenv("FC","mpiifort")

to gsi_hercules.intel.lua and gsi_orion.intel.lua.

DavidHuber-NOAA · 2025-08-15T16:50:06Z

I know the issue for the SFS forecasts now. I made a local change to config.ufs to increase layout_x and layout_y for the C96C48_hybatmDA test case and copied it to all tests. However, the GFS config.ufs is not compatible with SFS. I'm relaunching the C96mx100_S2S case now with a clean EXPDIR.

DavidHuber-NOAA · 2025-08-15T16:57:31Z

SFS forecasts have made it to the 12 hour mark without any signs of instability.

NeilBarton-NOAA · 2025-08-15T17:16:26Z

great. thanks @DavidHuber-NOAA

DavidHuber-NOAA · 2025-08-15T17:44:01Z

All tests passed on Hercules 🎉:

C48_ATM_192d
       CYCLE                    TASK                       JOBID               STATE         EXIT STATUS     TRIES      DURATION
================================================================================================================================
C48mx500_3DVarAOWCDA_192d
       CYCLE                    TASK                       JOBID               STATE         EXIT STATUS     TRIES      DURATION
================================================================================================================================
================================================================================================================================
C48mx500_hybAOWCDA_192d
       CYCLE                    TASK                       JOBID               STATE         EXIT STATUS     TRIES      DURATION
================================================================================================================================
================================================================================================================================
C48_S2SW_192d
       CYCLE                    TASK                       JOBID               STATE         EXIT STATUS     TRIES      DURATION
================================================================================================================================
C48_S2SWA_gefs_192d
       CYCLE                    TASK                       JOBID               STATE         EXIT STATUS     TRIES      DURATION
================================================================================================================================
C96_atm3DVar_192d
       CYCLE                    TASK                       JOBID               STATE         EXIT STATUS     TRIES      DURATION
================================================================================================================================
================================================================================================================================
================================================================================================================================
C96C48_hybatmDA_192d
       CYCLE                    TASK                       JOBID               STATE         EXIT STATUS     TRIES      DURATION
================================================================================================================================
================================================================================================================================
================================================================================================================================
C96C48mx500_S2SW_cyc_gfs_192d
       CYCLE                    TASK                       JOBID               STATE         EXIT STATUS     TRIES      DURATION
================================================================================================================================
================================================================================================================================
================================================================================================================================
================================================================================================================================
C96mx100_S2S_192e
       CYCLE                    TASK                       JOBID               STATE         EXIT STATUS     TRIES      DURATION
================================================================================================================================

DavidHuber-NOAA · 2025-08-15T17:44:20Z

Requesting final approvals for this PR.

CoryMartin-NOAA

JessicaMeixner-NOAA

Yay!!!!!

@bbakernoaa - thanks for helping get all the gocart model update parts together.

Huge thanks to @DavidHuber-NOAA for all the work on getting all of the spack-stack pieces together!!

aerorahul · 2025-08-15T17:59:28Z

🎉

aerorahul · 2025-08-15T18:00:53Z

Thank you all for your effort in getting this and several other issues resolved with this PR.

DavidHuber-NOAA added 16 commits May 27, 2025 13:30

Update gfs_utils to SS 1.9.1

d424b30

Merge branch 'feature/191' of github.com:davidhuber-noaa/global-workf…

dc4511b

…low into feature/191

Update gfs_utils branch

5736669

Use the GDASApp feature/spack-stack191 branch

e2baff0

Update GSI branch to include SS/191 and Ursa support

404019f

Update gsi_utils hash to include new CI library versions

d8bb7a7

Update gsi hash to follow develop

c016ad8

Update gsi hash

90cc23c

Update gsi_utils hash to include 191 support on Hera

eb3ae65

Add 1.9.1 support for gsi_monitor

6534456

Add support for 191 in the weather model

b9e3b39

Add spack-stack 1.9.1 support for the UPP

f593f52

Update ufs/gocart submodule (upstream)

7efdb1b

Merge develop, keep spack-stack 1.9.1 submodules

a47a1c3

Update gsi hash

3c81906

Merge remote-tracking branch 'emc/develop' into feature/191

e716eaf

DavidHuber-NOAA mentioned this pull request Jun 18, 2025

Update GDASApp to spack-stack/1.9.1 NOAA-EMC/GDASApp#1283

Closed

DavidHuber-NOAA added 5 commits June 18, 2025 14:27

Update gsi_utils for MSU

8e65475

Update gfs_utils for hera/hercules/wcoss2 support

cf9d3d0

Update gsi_utils for ci updates

b9860a0

Rename module loading script

09583c6

Convert hercules module to common loading principle; upgrade versions…

bfa52b4

… to 1.9.1

github-advanced-security AI found potential problems Jun 20, 2025

View reviewed changes

Comment thread ush/load_gw_run_modules.sh Fixed

Comment thread ush/load_gw_run_modules.sh Fixed

DavidHuber-NOAA added 7 commits June 20, 2025 10:10

Convert build.hercules.ver to common method

89a8bb7

Remove support for C5

c0c9994

Convert all RDHPCS version files to common methodology

81543ae

Rename modules

6cd766d

Rename ci modules

06904c8

Remove c5 modules

cd564a9

Update module names in gw_setup.sh

8ece674

JessicaMeixner-NOAA mentioned this pull request Aug 15, 2025

Port Global Workflow to RDHPCS Ursa #3674

Closed

17 tasks

CoryMartin-NOAA reviewed Aug 15, 2025

View reviewed changes

Comment thread dev/parm/config/gfs/config.resources.URSA

Allow more memory per node for aeroanlvar on Ursa

a0120ca

CoryMartin-NOAA approved these changes Aug 15, 2025

View reviewed changes

JessicaMeixner-NOAA reviewed Aug 15, 2025

View reviewed changes

This was referenced Aug 15, 2025

Adopt a standardized way to set versions and load modules in the global workflow #3809

Closed

Migrate Hera to /scratch3/4 #3874

Closed

DavidHuber-NOAA mentioned this pull request Sep 22, 2025

Update WCOSS2 Modules #2106

Closed

Conversation

DavidHuber-NOAA commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

Uh oh!

Uh oh!

DavidHuber-NOAA commented Aug 15, 2025

Uh oh!

JessicaMeixner-NOAA commented Aug 15, 2025

Uh oh!

NeilBarton-NOAA commented Aug 15, 2025

Uh oh!

DavidHuber-NOAA commented Aug 15, 2025

Uh oh!

NeilBarton-NOAA commented Aug 15, 2025

Uh oh!

DavidHuber-NOAA commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JessicaMeixner-NOAA commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DavidHuber-NOAA commented Aug 15, 2025

Uh oh!

CatherineThomas-NOAA commented Aug 15, 2025

Uh oh!

JessicaMeixner-NOAA commented Aug 15, 2025

Uh oh!

JessicaMeixner-NOAA commented Aug 15, 2025

Uh oh!

CoryMartin-NOAA commented Aug 15, 2025

Uh oh!

bbakernoaa commented Aug 15, 2025

Uh oh!

Uh oh!

DavidHuber-NOAA commented Aug 15, 2025

Uh oh!

RussTreadon-NOAA commented Aug 15, 2025

Uh oh!

DavidHuber-NOAA commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DavidHuber-NOAA commented Aug 15, 2025

Uh oh!

NeilBarton-NOAA commented Aug 15, 2025

Uh oh!

DavidHuber-NOAA commented Aug 15, 2025

Uh oh!

DavidHuber-NOAA commented Aug 15, 2025

Uh oh!

CoryMartin-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

JessicaMeixner-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

aerorahul commented Aug 15, 2025

Uh oh!

aerorahul commented Aug 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants

DavidHuber-NOAA commented Jun 13, 2025 •

edited

Loading

DavidHuber-NOAA commented Aug 15, 2025 •

edited

Loading

JessicaMeixner-NOAA commented Aug 15, 2025 •

edited

Loading

DavidHuber-NOAA commented Aug 15, 2025 •

edited

Loading