Skip to content

Update JEDI hashes (20250407)#1595

Merged
RussTreadon-NOAA merged 10 commits into
developfrom
feature/stable-nightly
Apr 7, 2025
Merged

Update JEDI hashes (20250407)#1595
RussTreadon-NOAA merged 10 commits into
developfrom
feature/stable-nightly

Conversation

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor

@RussTreadon-NOAA RussTreadon-NOAA commented Apr 5, 2025

Description

Weekly update of JEDI hashes

None

Issues

Resolves #1594

Automated CI tests to run in Global Workflow

  • atm_jjob
  • C96C48_ufs_hybatmDA
  • C96C48_hybatmaerosnowDA
  • C48mx500_3DVarAOWCDA
  • C48mx500_hybAOWCDA
  • C96C48_hybatmDA

@RussTreadon-NOAA RussTreadon-NOAA self-assigned this Apr 5, 2025
@RussTreadon-NOAA RussTreadon-NOAA added orion-GW-RT Queue for automated testing with global-workflow on Orion hercules-GW-RT Queue for automated testing with global-workflow on Hercules labels Apr 5, 2025
@emcbot emcbot added hercules-GW-RT-Running Automated testing with global-workflow running on Hercules orion-GW-RT-Running Automated testing with global-workflow running on Orion and removed hercules-GW-RT Queue for automated testing with global-workflow on Hercules orion-GW-RT Queue for automated testing with global-workflow on Orion labels Apr 5, 2025
@emcbot
Copy link
Copy Markdown

emcbot commented Apr 5, 2025

Automated GW-GDASApp Testing Results:
Machine: hercules

Start: Sat Apr  5 10:48:05 CDT 2025 on hercules-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sat Apr  5 11:24:10 CDT 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                  *Failed*
Tests: Failed at Sat Apr  5 12:11:31 CDT 2025
Tests: 93% tests passed, 9 tests failed out of 136
	2070 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_prepoceanobs_202103250000 (Failed)
	2073 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlvar_202103250000 (Failed)
	2074 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlchkpt_202103250000 (Failed)
	2075 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlfinal_202103250000 (Failed)
	2081 - test_gdasapp_C48mx500_hybAOWCDA_gdas_prepoceanobs_202103250000 (Failed)
	2083 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlletkf_202103250000 (Failed)
	2085 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlvar_202103250000 (Failed)
	2087 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlchkpt_202103250000 (Failed)
	2088 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlfinal_202103250000 (Failed)
Tests: see output at /work2/noaa/da/role-da/CI/hercules/GDASApp/workflow/PR/1595/global-workflow/sorc/gdas.cd/build/log.ctest

@emcbot emcbot added hercules-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hercules-GW-RT-Running Automated testing with global-workflow running on Hercules labels Apr 5, 2025
@emcbot
Copy link
Copy Markdown

emcbot commented Apr 5, 2025

Automated GW-GDASApp Testing Results:
Machine: orion

Start: Sat Apr  5 10:48:48 AM CDT 2025 on orion-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sat Apr  5 11:53:03 AM CDT 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                  *Failed*
Tests: Failed at Sat Apr  5 01:45:17 PM CDT 2025
Tests: 93% tests passed, 9 tests failed out of 136
	2070 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_prepoceanobs_202103250000 (Failed)
	2073 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlvar_202103250000 (Failed)
	2074 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlchkpt_202103250000 (Failed)
	2075 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlfinal_202103250000 (Failed)
	2081 - test_gdasapp_C48mx500_hybAOWCDA_gdas_prepoceanobs_202103250000 (Failed)
	2083 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlletkf_202103250000 (Failed)
	2085 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlvar_202103250000 (Failed)
	2087 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlchkpt_202103250000 (Failed)
	2088 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlfinal_202103250000 (Failed)
Tests: see output at /work2/noaa/da/role-da/CI/orion/GDASApp/workflow/PR/1595/global-workflow/sorc/gdas.cd/build/log.ctest

@emcbot emcbot added orion-GW-RT-Failed Automated testing with global-workflow failed on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Apr 5, 2025
@RussTreadon-NOAA RussTreadon-NOAA added the hera-GW-RT Queue for automated testing with global-workflow on Hera label Apr 5, 2025
@emcbot emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hera-GW-RT Queue for automated testing with global-workflow on Hera labels Apr 5, 2025
@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor Author

WCOSS2 g-w CI

Install feature/stable-nightly at 53276d4 in g-w develop at 9f3615d on Dogwood. Run g-w CI with the following results

/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_ATM_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Apr 05 2025 17:30:33    Apr 05 2025 19:05:26
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_3DVarAOWCDA_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Apr 05 2025 17:30:35    Apr 05 2025 17:50:35
202103250000        Done    Apr 05 2025 17:30:35    Apr 05 2025 19:35:25
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_hybAOWCDA_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Apr 05 2025 17:30:37    Apr 05 2025 17:50:40
202103250000        Done    Apr 05 2025 17:30:37    Apr 05 2025 19:05:41
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_S2SWA_gefs_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Apr 05 2025 17:30:40    Apr 05 2025 19:30:42
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_S2SW_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Apr 05 2025 17:30:46    Apr 05 2025 19:05:51
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96_atm3DVar_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Apr 05 2025 17:30:48    Apr 05 2025 17:50:55
202112210000        Done    Apr 05 2025 17:30:48    Apr 05 2025 20:05:35
202112210600        Done    Apr 05 2025 17:30:48    Apr 05 2025 19:45:33
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmaerosnowDA_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Apr 05 2025 17:30:50    Apr 05 2025 17:55:59
202112201800        Done    Apr 05 2025 17:30:50    Apr 05 2025 20:00:36
202112210000        Done    Apr 05 2025 17:30:50    Apr 05 2025 19:55:36
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmDA_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Apr 05 2025 17:30:52    Apr 05 2025 17:51:03
202112210000        Done    Apr 05 2025 17:30:52    Apr 05 2025 19:35:42
202112210600        Done    Apr 05 2025 17:30:52    Apr 05 2025 19:45:41
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_ufs_hybatmDA_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    Apr 05 2025 17:30:54    Apr 05 2025 17:51:08
202402240000        Done    Apr 05 2025 17:30:54    Apr 05 2025 20:30:49
202402240600        Done    Apr 05 2025 17:30:54    Apr 05 2025 20:25:38
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96mx100_S2S_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
199405010000    sfs_fcst_mem000_seg0                    28816198                DEAD                 271         2         110.0
199405010000    sfs_fcst_mem001_seg0                    28816199                DEAD                 271         2         110.0
199405010000    sfs_fcst_mem002_seg0                    28816200                DEAD                 271         2         110.0

All g-w CI cases pass except C96mx100_S2S. This is a known failure - see g-w issue #3531. The C96mx100_S2S case does not exercise GDASApp.

@emcbot
Copy link
Copy Markdown

emcbot commented Apr 5, 2025

Automated GW-GDASApp Testing Results:
Machine: hera

Start: Sat Apr  5 20:58:57 UTC 2025 on hfe08
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sat Apr  5 21:42:51 UTC 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Sat Apr  5 23:06:56 UTC 2025
Tests: 100% tests passed, 0 tests failed out of 136

@emcbot emcbot added hera-GW-RT-Passed Automated testing with global-workflow successful on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Apr 5, 2025
@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor Author

The gdas_prepoceanobs failures on Orion and Hercules are for the same reason in the 3DVarAOWCDA and hybAOWCDA g-w CI cases. Below is the failure from the Orion C48mx500_hybAOWCDA gdas_prepoceanobs

+ setpdy.sh[49]'[' -z 20210325 ']'
+ setpdy.sh[53]sed 's/[0-9]\{8\}/20210325/' /work2/noaa/da/role-da/CI/orion/GDASApp/workflow/PR/1595/global-workflow/sorc/gdas.cd/build/gdas/test/gw-ci/../../test/gw-ci/C48mx500_hybAOWCDA/COMROOT/date/t00z
sed: can't read /work2/noaa/da/role-da/CI/orion/GDASApp/workflow/PR/1595/global-workflow/sorc/gdas.cd/build/gdas/test/gw-ci/../../test/gw-ci/C48mx500_hybAOWCDA/COMROOT/date/t00z: No such file or directory
++ setpdy.sh[58]finddate.sh 20210325 s-7
+++ finddate.sh[27]set +x
/apps/contrib/NCEP/libs/hpc-stack/intel-2018.4/prod_util/1.2.2/bin/finddate.sh: line 263: pdstr: unbound variable
+ setpdy.sh[58]DATElne_minus=
+ setpdy.sh[59]export err=1
+ setpdy.sh[59]err=1
+ setpdy.sh[59]err_chk
+ err_chk[14]COREROOT=/gpfs/hps/ptmp/production.core/prepoceanobs.541622
+ err_chk[16]'[' -z 1 ']'
+ err_chk[18]'[' 1 -ne 0 ']'
+ err_chk[21]'[' 1 -eq 139 ']'
+ err_chk[33]err_exit
+ environment[0]set +eux

    -------------------------------------------------------------
    -- FATAL ERROR: Job prepoceanobs.541622 failed RETURN CODE 1
    -- ABNORMAL EXIT at Sat Apr  5 12:04:17 CDT 2025 on orion-06-64
    -------------------------------------------------------------

It's not clear to me why this job runs to completion on Hera and Dogwood but fail on Hercules and Orion.

The g-w develop used on Hercules, Orion, Hera, and Dogwood is 9f3615d. The hash is associated with g-w PR #3515. This PR mentions failures in gdas_prepoceanobs. It's not clear from the PR how the setpdy problem was fixed.

Tagging @DavidHuber-NOAA , @guillaumevernieres

@RussTreadon-NOAA RussTreadon-NOAA added the hercules-GW-RT Queue for automated testing with global-workflow on Hercules label Apr 7, 2025
@emcbot emcbot added orion-GW-RT-Running Automated testing with global-workflow running on Orion hercules-GW-RT-Running Automated testing with global-workflow running on Hercules hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed orion-GW-RT Queue for automated testing with global-workflow on Orion hercules-GW-RT Queue for automated testing with global-workflow on Hercules hera-GW-RT Queue for automated testing with global-workflow on Hera labels Apr 7, 2025
@emcbot
Copy link
Copy Markdown

emcbot commented Apr 7, 2025

Automated GW-GDASApp Testing Results:
Machine: hercules

Start: Mon Apr  7 06:52:37 CDT 2025 on hercules-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon Apr  7 07:29:24 CDT 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Mon Apr  7 08:16:17 CDT 2025
Tests: 100% tests passed, 0 tests failed out of 136

@emcbot emcbot added hercules-GW-RT-Passed Automated testing with global-workflow successful on Hercules and removed hercules-GW-RT-Running Automated testing with global-workflow running on Hercules labels Apr 7, 2025
Copy link
Copy Markdown
Contributor

@guillaumevernieres guillaumevernieres left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Copy link
Copy Markdown
Collaborator

@DavidHuber-NOAA DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for finding the fix @RussTreadon-NOAA!

@emcbot
Copy link
Copy Markdown

emcbot commented Apr 7, 2025

Automated GW-GDASApp Testing Results:
Machine: hera

Start: Mon Apr  7 11:57:54 UTC 2025 on hfe05
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon Apr  7 12:40:16 UTC 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Mon Apr  7 13:51:10 UTC 2025
Tests: 100% tests passed, 0 tests failed out of 136

@emcbot emcbot added hera-GW-RT-Passed Automated testing with global-workflow successful on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Apr 7, 2025
@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor Author

Once this PR is merged into GDASApp develop, we can close g-w issue #3555

@DavidHuber-NOAA
Copy link
Copy Markdown
Collaborator

@RussTreadon-NOAA I think we will also need to update the GDASApp hash in the global workflow, correct?

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor Author

True, I'm thinking from a GDASApp-centric perspective. GDASApp PR #1595 can run g-w CI but g-w can not run GDASApp without an updated sorc/gdas.cd hash.

@emcbot
Copy link
Copy Markdown

emcbot commented Apr 7, 2025

Automated GW-GDASApp Testing Results:
Machine: orion

Start: Mon Apr  7 06:52:44 AM CDT 2025 on orion-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon Apr  7 07:50:40 AM CDT 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Mon Apr  7 09:28:54 AM CDT 2025
Tests: 100% tests passed, 0 tests failed out of 136

@emcbot emcbot added orion-GW-RT-Passed Automated testing with global-workflow successful on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Apr 7, 2025
@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor Author

GDASApp tests pass on Hera, Hercules, and Orion. g-w CI passes on Dogwood. Changes reviewed and approved. Merge into develop.

@RussTreadon-NOAA RussTreadon-NOAA merged commit c3d0cab into develop Apr 7, 2025
@RussTreadon-NOAA RussTreadon-NOAA deleted the feature/stable-nightly branch April 7, 2025 14:33
DavidHuber-NOAA pushed a commit to NOAA-EMC/global-workflow that referenced this pull request Apr 8, 2025
# Description

This PR updates the `sorc/gdas.cd` hash to resolve failures in marine DA
job `gdas_prepoceanobs` on Hercules and Orion. The failure is due to the
wrong `prod_util` module being used on MSU machines. GDASApp PR
[#1595](NOAA-EMC/GDASApp#1595) updated the
Hercules and Orion modules to use the correct `prod_util`. Marine DA job
`gdas_prepoceanobs` runs to completion with the updated `sorc/gdas.cd`
hash.

  Resolves #3555

# Type of change
- [x] Bug fix (fixes something broken)


# Change characteristics
- Is this a breaking change (a change in existing functionality)? **NO**
- Does this change require a documentation update? **NO**
- Does this change require an update to any of the following submodules?
**NO**

# How has this been tested?
GDASApp based g-w CI was run on Hercules and Orion as part of GDASApp PR
[#1595](NOAA-EMC/GDASApp#1595) testing. All
tests, including `gdas_prepoceanobs`, pass on MSU machines.

# Checklist
- [x] Any dependent changes have been merged and published
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my own code
- [ ] New and existing tests pass with my changes
jspsl pushed a commit to NOAA-PSL/GDASApp that referenced this pull request Apr 30, 2025
DavidNew-NOAA pushed a commit that referenced this pull request Jan 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hera-GW-RT-Passed Automated testing with global-workflow successful on Hera hercules-GW-RT-Passed Automated testing with global-workflow successful on Hercules orion-GW-RT-Passed Automated testing with global-workflow successful on Orion

Projects

None yet

Development

Successfully merging this pull request may close these issues.

20250405 stable nightly failure

5 participants