Skip to content

Update JEDI hashes (20250324)#1567

Merged
RussTreadon-NOAA merged 3 commits into
developfrom
feature/stable-nightly
Mar 24, 2025
Merged

Update JEDI hashes (20250324)#1567
RussTreadon-NOAA merged 3 commits into
developfrom
feature/stable-nightly

Conversation

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor

@RussTreadon-NOAA RussTreadon-NOAA commented Mar 23, 2025

Description

Weekly update of JEDI hashes

Companion PRs

None

Issues

None

Automated CI tests to run in Global Workflow

  • atm_jjob
  • C96C48_ufs_hybatmDA
  • C96C48_hybatmaerosnowDA
  • C48mx500_3DVarAOWCDA
  • C48mx500_hybAOWCDA
  • C96C48_hybatmDA

@RussTreadon-NOAA RussTreadon-NOAA self-assigned this Mar 23, 2025
@RussTreadon-NOAA RussTreadon-NOAA added hera-GW-RT Queue for automated testing with global-workflow on Hera hercules-GW-RT Queue for automated testing with global-workflow on Hercules orion-GW-RT Queue for automated testing with global-workflow on Orion labels Mar 23, 2025
@emcbot emcbot added hercules-GW-RT-Running Automated testing with global-workflow running on Hercules orion-GW-RT-Running Automated testing with global-workflow running on Orion hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hercules-GW-RT Queue for automated testing with global-workflow on Hercules orion-GW-RT Queue for automated testing with global-workflow on Orion hera-GW-RT Queue for automated testing with global-workflow on Hera labels Mar 23, 2025
@emcbot
Copy link
Copy Markdown

emcbot commented Mar 23, 2025

Automated GW-GDASApp Testing Results:
Machine: hercules

Start: Sun Mar 23 07:53:24 CDT 2025 on hercules-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sun Mar 23 08:31:35 CDT 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                  *Failed*
Tests: Failed at Sun Mar 23 14:43:07 CDT 2025
Tests: 38% tests passed, 85 tests failed out of 136
	1995 - test_gdasapp_snow_letkfoi_snowda (Failed)
	1999 - test_gdasapp_C96C48_hybatmDA_gdas_fcst_202112201800 (Failed)
	2000 - test_gdasapp_C96C48_hybatmDA_gdas_atmos_prod_202112201800 (Failed)
	2002 - test_gdasapp_C96C48_hybatmDA_enkfgdas_fcst_202112201800 (Failed)
	2003 - test_gdasapp_C96C48_hybatmDA_enkfgdas_echgres_202112201800 (Failed)
	2004 - test_gdasapp_C96C48_hybatmDA_enkfgdas_epmn_202112201800 (Failed)
	2005 - test_gdasapp_C96C48_hybatmDA_gdas_prep_202112210000 (Failed)
	2006 - test_gdasapp_C96C48_hybatmDA_gdas_anal_202112210000 (Failed)
	2007 - test_gdasapp_C96C48_hybatmDA_gdas_sfcanl_202112210000 (Failed)
	2008 - test_gdasapp_C96C48_hybatmDA_gdas_analcalc_202112210000 (Failed)
	2009 - test_gdasapp_C96C48_hybatmDA_gdas_fcst_202112210000 (Failed)
	2010 - test_gdasapp_C96C48_hybatmDA_enkfgdas_eobs_202112210000 (Failed)
	2011 - test_gdasapp_C96C48_hybatmDA_enkfgdas_ediag_202112210000 (Failed)
	2012 - test_gdasapp_C96C48_hybatmDA_enkfgdas_eupd_202112210000 (Failed)
	2013 - test_gdasapp_C96C48_hybatmDA_enkfgdas_ecmn_202112210000 (Failed)
	2014 - test_gdasapp_C96C48_hybatmDA_enkfgdas_esfc_202112210000 (Failed)
	2015 - test_gdasapp_C96C48_hybatmDA_enkfgdas_fcst_202112210000 (Failed)
	2018 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_fcst_202402231800 (Failed)
	2019 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmos_prod_202402231800 (Failed)
	2021 - test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_fcst_202402231800 (Failed)
	2022 - test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_epmn_202402231800 (Failed)
	2023 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_prep_202402240000 (Failed)
	2024 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_prepatmiodaobs_202402240000 (Failed)
	2025 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlinit_202402240000 (Failed)
	2026 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlvar_202402240000 (Failed)
	2027 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlfv3inc_202402240000 (Failed)
	2028 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlfinal_202402240000 (Failed)
	2029 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_sfcanl_202402240000 (Failed)
	2030 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_analcalc_fv3jedi_202402240000 (Failed)
	2031 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_fcst_202402240000 (Failed)
	2032 - test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlinit_202402240000 (Failed)
	2033 - test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlobs_202402240000 (Failed)
	2034 - test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlsol_202402240000 (Failed)
	2035 - test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlfv3inc_202402240000 (Failed)
	2036 - test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlfinal_202402240000 (Failed)
	2037 - test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_ecen_fv3jedi_202402240000 (Failed)
	2038 - test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_esfc_202402240000 (Failed)
	2039 - test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_fcst_202402240000 (Failed)
	2042 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_fcst_202112201200 (Failed)
	2043 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_atmos_prod_202112201200 (Failed)
	2044 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_aeroanlgenb_202112201200 (Failed)
	2046 - test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_fcst_202112201200 (Failed)
	2047 - test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_echgres_202112201200 (Failed)
	2048 - test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_epmn_202112201200 (Failed)
	2049 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_prep_202112201800 (Failed)
	2050 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_anal_202112201800 (Failed)
	2051 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_aeroanlinit_202112201800 (Failed)
	2052 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_aeroanlvar_202112201800 (Failed)
	2053 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_aeroanlfinal_202112201800 (Failed)
	2054 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_snowanl_202112201800 (Failed)
	2055 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_sfcanl_202112201800 (Failed)
	2056 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_analcalc_202112201800 (Failed)
	2057 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_fcst_202112201800 (Failed)
	2058 - test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_eobs_202112201800 (Failed)
	2059 - test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_ediag_202112201800 (Failed)
	2060 - test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_eupd_202112201800 (Failed)
	2061 - test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_ecmn_202112201800 (Failed)
	2062 - test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_esnowanl_202112201800 (Failed)
	2063 - test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_esfc_202112201800 (Failed)
	2064 - test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_fcst_202112201800 (Failed)
	2068 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_fcst_202103241800 (Failed)
	2069 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_prepoceanobs_202103250000 (Failed)
	2070 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marinebmat_202103250000 (Failed)
	2071 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlinit_202103250000 (Failed)
	2072 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlvar_202103250000 (Failed)
	2073 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlchkpt_202103250000 (Failed)
	2074 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlfinal_202103250000 (Failed)
	2077 - test_gdasapp_C48mx500_hybAOWCDA_gdas_fcst_202103241800 (Failed)
	2079 - test_gdasapp_C48mx500_hybAOWCDA_enkfgdas_fcst_202103241800 (Failed)
	2080 - test_gdasapp_C48mx500_hybAOWCDA_gdas_prepoceanobs_202103250000 (Failed)
	2081 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marinebmat_202103250000 (Failed)
	2082 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlletkf_202103250000 (Failed)
	2083 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlinit_202103250000 (Failed)
	2084 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlvar_202103250000 (Failed)
	2085 - test_gdasapp_C48mx500_hybAOWCDA_gdas_ocnanalecen_202103250000 (Failed)
	2086 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlchkpt_202103250000 (Failed)
	2087 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlfinal_202103250000 (Failed)
	2091 - test_gdasapp_atm_jjob_var_init (Failed)
	2095 - test_gdasapp_atm_jjob_ens_init (Failed)
	2096 - test_gdasapp_atm_jjob_ens_letkf (Failed)
	2097 - test_gdasapp_atm_jjob_ens_init_split (Failed)
	2098 - test_gdasapp_atm_jjob_ens_obs (Failed)
	2099 - test_gdasapp_atm_jjob_ens_sol (Failed)
	2100 - test_gdasapp_atm_jjob_ens_inc (Failed)
	2101 - test_gdasapp_atm_jjob_ens_final (Failed)
Tests: see output at /work2/noaa/da/role-da/CI/hercules/GDASApp/workflow/PR/1567/global-workflow/sorc/gdas.cd/build/log.ctest

@emcbot emcbot added hercules-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hercules-GW-RT-Running Automated testing with global-workflow running on Hercules labels Mar 23, 2025
@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor Author

Hercules ctests

A check of the Hercules failures finds numerous instances of jobs being cancelled because the job reached the specified wall clock limit

gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122018/gdas_snowanl.log:slurmstepd: error: *** JOB 4864852 ON hercules-02-39 CANCELLED AT 2025-03-23T10:31:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122018/gdas_aeroanlinit.log:slurmstepd: error: *** JOB 4864903 ON hercules-08-55 CANCELLED AT 2025-03-23T11:46:42 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122018/enkfgdas_fcst_mem001.log:slurmstepd: error: *** JOB 4865218 ON hercules-08-02 CANCELLED AT 2025-03-23T13:26:43 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/gdas_aeroanlgenb.log:slurmstepd: error: *** JOB 4864631 ON hercules-04-05 CANCELLED AT 2025-03-23T10:36:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/gdas_atmos_prod_f003.log:slurmstepd: error: *** JOB 4864635 ON hercules-02-48 CANCELLED AT 2025-03-23T09:46:40 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/enkfgdas_fcst_mem001.log.0: 0: slurmstepd: error: *** STEP 4864495.0 ON hercules-04-17 CANCELLED AT 2025-03-23T09:01:39 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/enkfgdas_fcst_mem001.log.0:slurmstepd: error: *** JOB 4864495 ON hercules-04-17 CANCELLED AT 2025-03-23T09:01:39 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/enkfgdas_fcst_mem002.log.0:slurmstepd: error: *** JOB 4864496 ON hercules-04-18 CANCELLED AT 2025-03-23T09:01:39 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/enkfgdas_epos001.log: 0: slurmstepd: error: *** STEP 4864782.0 ON hercules-08-07 CANCELLED AT 2025-03-23T09:51:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/enkfgdas_epos001.log:slurmstepd: error: *** JOB 4864782 ON hercules-08-07 CANCELLED AT 2025-03-23T09:51:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/enkfgdas_epos000.log: 0: slurmstepd: error: *** STEP 4864781.0 ON hercules-08-01 CANCELLED AT 2025-03-23T09:51:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/enkfgdas_epos000.log:slurmstepd: error: *** JOB 4864781 ON hercules-08-01 CANCELLED AT 2025-03-23T09:51:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/enkfgdas_epos003.log: 0: slurmstepd: error: *** STEP 4864784.0 ON hercules-07-61 CANCELLED AT 2025-03-23T10:01:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/enkfgdas_epos003.log:slurmstepd: error: *** JOB 4864784 ON hercules-07-61 CANCELLED AT 2025-03-23T10:01:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/gdas_fcst_seg0.log: 0: slurmstepd: error: *** STEP 4864534.0 ON hercules-06-63 CANCELLED AT 2025-03-23T09:21:40 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/gdas_fcst_seg0.log:slurmstepd: error: *** JOB 4864534 ON hercules-06-63 CANCELLED AT 2025-03-23T09:21:40 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/enkfgdas_epos002.log: 0: slurmstepd: error: *** STEP 4864783.0 ON hercules-06-06 CANCELLED AT 2025-03-23T09:56:40 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/enkfgdas_epos002.log:slurmstepd: error: *** JOB 4864783 ON hercules-06-06 CANCELLED AT 2025-03-23T09:56:40 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/enkfgdas_fcst_mem002.log: 0: slurmstepd: error: *** STEP 4864591.0 ON hercules-06-03 CANCELLED AT 2025-03-23T09:31:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/enkfgdas_fcst_mem002.log:slurmstepd: error: *** JOB 4864591 ON hercules-06-03 CANCELLED AT 2025-03-23T09:31:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/enkfgdas_fcst_mem001.log: 0: slurmstepd: error: *** STEP 4864590.0 ON hercules-06-02 CANCELLED AT 2025-03-23T09:31:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/enkfgdas_fcst_mem001.log:slurmstepd: error: *** JOB 4864590 ON hercules-06-02 CANCELLED AT 2025-03-23T09:31:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/gdas_fcst_seg0.log.0: 0: slurmstepd: error: *** STEP 4864487.0 ON hercules-05-28 CANCELLED AT 2025-03-23T08:56:40 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/gdas_fcst_seg0.log.0:slurmstepd: error: *** JOB 4864487 ON hercules-05-28 CANCELLED AT 2025-03-23T08:56:40 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022400/enkfgdas_atmensanlinit.log:slurmstepd: error: *** JOB 4864917 ON hercules-07-58 CANCELLED AT 2025-03-23T11:56:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022400/gdas_atmanlinit.log:slurmstepd: error: *** JOB 4864925 ON hercules-05-06 CANCELLED AT 2025-03-23T12:01:42 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022400/enkfgdas_atmensanlfinal.log:slurmstepd: error: *** JOB 4865237 ON hercules-08-02 CANCELLED AT 2025-03-23T14:11:44 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022400/enkfgdas_atmensanlsol.log:slurmstepd: error: *** JOB 4865185 ON hercules-03-35 CANCELLED AT 2025-03-23T13:31:44 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022400/gdas_analcalc_fv3jedi.log:slurmstepd: error: *** JOB 4865266 ON hercules-07-05 CANCELLED AT 2025-03-23T14:26:43 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022400/gdas_prepatmiodaobs.log:slurmstepd: error: *** JOB 4864811 ON hercules-04-46 CANCELLED AT 2025-03-23T10:41:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022400/gdas_atmanlfinal.log:slurmstepd: error: *** JOB 4865236 ON hercules-07-05 CANCELLED AT 2025-03-23T14:06:44 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022318/enkfgdas_fcst_mem001.log.0: 0: slurmstepd: error: *** STEP 4864497.0 ON hercules-04-19 CANCELLED AT 2025-03-23T09:01:39 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022318/enkfgdas_fcst_mem001.log.0:slurmstepd: error: *** JOB 4864497 ON hercules-04-19 CANCELLED AT 2025-03-23T09:01:39 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022318/enkfgdas_fcst_mem002.log.0:slurmstepd: error: *** JOB 4864498 ON hercules-04-20 CANCELLED AT 2025-03-23T09:01:39 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022318/enkfgdas_epos001.log: 0: slurmstepd: error: *** STEP 4864769.0 ON hercules-08-49 CANCELLED AT 2025-03-23T09:46:40 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022318/enkfgdas_epos001.log:slurmstepd: error: *** JOB 4864769 ON hercules-08-49 CANCELLED AT 2025-03-23T09:46:40 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022318/enkfgdas_epos000.log: 0: slurmstepd: error: *** STEP 4864768.0 ON hercules-06-62 CANCELLED AT 2025-03-23T09:46:40 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022318/enkfgdas_epos000.log:slurmstepd: error: *** JOB 4864768 ON hercules-06-62 CANCELLED AT 2025-03-23T09:46:40 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022318/enkfgdas_epos003.log: 0: slurmstepd: error: *** STEP 4864772.0 ON hercules-07-58 CANCELLED AT 2025-03-23T09:46:40 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022318/enkfgdas_epos003.log:slurmstepd: error: *** JOB 4864772 ON hercules-07-58 CANCELLED AT 2025-03-23T09:46:40 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022318/gdas_fcst_seg0.log: 0: slurmstepd: error: *** STEP 4864490.0 ON hercules-02-07 CANCELLED AT 2025-03-23T09:01:39 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022318/gdas_fcst_seg0.log:slurmstepd: error: *** JOB 4864490 ON hercules-02-07 CANCELLED AT 2025-03-23T09:01:39 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022318/enkfgdas_epos002.log: 0: slurmstepd: error: *** STEP 4864771.0 ON hercules-08-50 CANCELLED AT 2025-03-23T09:46:40 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022318/enkfgdas_epos002.log:slurmstepd: error: *** JOB 4864771 ON hercules-08-50 CANCELLED AT 2025-03-23T09:46:40 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022318/enkfgdas_fcst_mem002.log:slurmstepd: error: *** JOB 4864568 ON hercules-01-22 CANCELLED AT 2025-03-23T09:31:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022318/enkfgdas_fcst_mem001.log:slurmstepd: error: *** JOB 4864567 ON hercules-07-10 CANCELLED AT 2025-03-23T09:26:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022318/enkfgdas_fcst_mem001.log: 0: slurmstepd: error: *** STEP 4864567.0 ON hercules-07-10 CANCELLED AT 2025-03-23T09:26:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmDA/COMROOT/C96C48_hybatmDA/logs/2021122018/gdas_fcst_seg0.log:slurmstepd: error: *** JOB 4864488 ON hercules-06-18 CANCELLED AT 2025-03-23T08:56:40 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmDA/COMROOT/C96C48_hybatmDA/logs/2021122018/enkfgdas_fcst_mem002.log:slurmstepd: error: *** JOB 4864492 ON hercules-02-09 CANCELLED AT 2025-03-23T09:01:39 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmDA/COMROOT/C96C48_hybatmDA/logs/2021122018/enkfgdas_fcst_mem001.log: 0: slurmstepd: error: *** STEP 4864491.0 ON hercules-02-08 CANCELLED AT 2025-03-23T09:01:39 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmDA/COMROOT/C96C48_hybatmDA/logs/2021122018/enkfgdas_fcst_mem001.log:slurmstepd: error: *** JOB 4864491 ON hercules-02-08 CANCELLED AT 2025-03-23T09:01:39 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_hybAOWCDA/COMROOT/C48mx500_hybAOWCDA/logs/2021032418/enkfgdas_fcst_mem001.log.0: 0: slurmstepd: error: *** STEP 4864501.0 ON hercules-04-21 CANCELLED AT 2025-03-23T09:01:39 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_hybAOWCDA/COMROOT/C48mx500_hybAOWCDA/logs/2021032418/enkfgdas_fcst_mem001.log.0:slurmstepd: error: *** JOB 4864501 ON hercules-04-21 CANCELLED AT 2025-03-23T09:01:39 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_hybAOWCDA/COMROOT/C48mx500_hybAOWCDA/logs/2021032418/enkfgdas_fcst_mem002.log.0:slurmstepd: error: *** JOB 4864502 ON hercules-02-42 CANCELLED AT 2025-03-23T09:01:39 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_hybAOWCDA/COMROOT/C48mx500_hybAOWCDA/logs/2021032418/gdas_fcst_seg0.log:slurmstepd: error: *** JOB 4864494 ON hercules-02-11 CANCELLED AT 2025-03-23T09:01:39 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_hybAOWCDA/COMROOT/C48mx500_hybAOWCDA/logs/2021032418/enkfgdas_fcst_mem002.log:slurmstepd: error: *** JOB 4864580 ON hercules-05-47 CANCELLED AT 2025-03-23T09:31:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_hybAOWCDA/COMROOT/C48mx500_hybAOWCDA/logs/2021032418/enkfgdas_fcst_mem001.log: 0: slurmstepd: error: *** STEP 4864579.0 ON hercules-06-55 CANCELLED AT 2025-03-23T09:31:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_hybAOWCDA/COMROOT/C48mx500_hybAOWCDA/logs/2021032418/enkfgdas_fcst_mem001.log:slurmstepd: error: *** JOB 4864579 ON hercules-06-55 CANCELLED AT 2025-03-23T09:31:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_hybAOWCDA/COMROOT/C48mx500_hybAOWCDA/logs/2021032500/gdas_ocnanalecen.log:slurmstepd: error: *** JOB 4865194 ON hercules-07-45 CANCELLED AT 2025-03-23T12:51:44 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_hybAOWCDA/COMROOT/C48mx500_hybAOWCDA/logs/2021032500/gdas_marineanlinit.log:slurmstepd: error: *** JOB 4864881 ON hercules-08-51 CANCELLED AT 2025-03-23T10:36:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_hybAOWCDA/COMROOT/C48mx500_hybAOWCDA/logs/2021032500/gdas_marinebmat.log:slurmstepd: error: *** JOB 4864778 ON hercules-04-51 CANCELLED AT 2025-03-23T10:21:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_hybAOWCDA/COMROOT/C48mx500_hybAOWCDA/logs/2021032500/gdas_prepoceanobs.log:slurmstepd: error: *** JOB 4864582 ON hercules-02-11 CANCELLED AT 2025-03-23T09:16:39 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_3DVarAOWCDA/COMROOT/C48mx500_3DVarAOWCDA/logs/2021032500/gdas_marineanlinit.log:slurmstepd: error: *** JOB 4864854 ON hercules-02-63 CANCELLED AT 2025-03-23T10:26:41 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_3DVarAOWCDA/COMROOT/C48mx500_3DVarAOWCDA/logs/2021032500/gdas_marineanlvar.log: 0: slurmstepd: error: *** STEP 4864886.0 ON hercules-01-22 CANCELLED AT 2025-03-23T12:16:42 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_3DVarAOWCDA/COMROOT/C48mx500_3DVarAOWCDA/logs/2021032500/gdas_marineanlvar.log:slurmstepd: error: *** JOB 4864886 ON hercules-01-22 CANCELLED AT 2025-03-23T12:16:42 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_3DVarAOWCDA/COMROOT/C48mx500_3DVarAOWCDA/logs/2021032500/gdas_marinebmat.log:slurmstepd: error: *** JOB 4864605 ON hercules-01-50 CANCELLED AT 2025-03-23T10:11:42 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_3DVarAOWCDA/COMROOT/C48mx500_3DVarAOWCDA/logs/2021032500/gdas_prepoceanobs.log:slurmstepd: error: *** JOB 4864599 ON hercules-04-03 CANCELLED AT 2025-03-23T09:21:40 DUE TO TIME LIMIT ***
gdas/test/atm/global-workflow/testrun/atmanlinit-4864500.out:slurmstepd: error: *** JOB 4864500 ON hercules-01-08 CANCELLED AT 2025-03-23T08:51:39 DUE TO TIME LIMIT ***
gdas/test/atm/global-workflow/testrun/atmensanlinit-4864484.out:slurmstepd: error: *** JOB 4864484 ON hercules-01-08 CANCELLED AT 2025-03-23T08:46:40 DUE TO TIME LIMIT ***
gdas/test/atm/global-workflow/testrun/atmensanlletkf-4864514.out:slurmstepd: error: *** JOB 4864514 ON hercules-08-11 CANCELLED AT 2025-03-23T09:21:40 DUE TO TIME LIMIT ***
gdas/test/atm/global-workflow/testrun/atmensanlobs-4864795.out:slurmstepd: error: *** JOB 4864795 ON hercules-04-33 CANCELLED AT 2025-03-23T10:07:11 DUE TO TIME LIMIT ***
gdas/test/atm/global-workflow/testrun/atmanlfv3inc-4864597.out:slurmstepd: error: *** JOB 4864597 ON hercules-01-06 CANCELLED AT 2025-03-23T09:21:40 DUE TO TIME LIMIT ***
gdas/test/atm/global-workflow/testrun/atmensanlsol-4864859.out:slurmstepd: error: *** JOB 4864859 ON hercules-01-09 CANCELLED AT 2025-03-23T10:46:11 DUE TO TIME LIMIT ***
gdas/test/atm/global-workflow/testrun/atmanlvar-4864523.out:slurmstepd: error: *** JOB 4864523 ON hercules-01-06 CANCELLED AT 2025-03-23T09:06:40 DUE TO TIME LIMIT ***
gdas/test/atm/global-workflow/testrun/atmanlfinal-4864630.out:slurmstepd: error: *** JOB 4864630 ON hercules-01-08 CANCELLED AT 2025-03-23T09:36:41 DUE TO TIME LIMIT ***
gdas/test/atm/global-workflow/testrun/atmensanlinit_split-4864636.out:slurmstepd: error: *** JOB 4864636 ON hercules-01-58 CANCELLED AT 2025-03-23T09:36:41 DUE TO TIME LIMIT ***

All these jobs run in /work2/noaa. It looks like the /work2 fileset is experiening IO issues. A MSU ticket will be opened.

@emcbot
Copy link
Copy Markdown

emcbot commented Mar 23, 2025

Automated GW-GDASApp Testing Results:
Machine: hera

Start: Sun Mar 23 12:55:27 UTC 2025 on hfe02
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sun Mar 23 13:36:00 UTC 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Sun Mar 23 20:03:47 UTC 2025
Tests: 100% tests passed, 0 tests failed out of 136

@emcbot emcbot added hera-GW-RT-Passed Automated testing with global-workflow successful on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Mar 23, 2025
@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor Author

Orion ctests

GDASApp ctests running on Orion are also failing to due jobs exceeding the specified wall clock limit. Here are examples of this failure from log file in /work2/noaa/da/role-da/CI/orion/GDASApp/workflow/PR/1567/global-workflow/sorc/gdas.cd/build

gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/enkfgdas_fcst_mem002.log.0:slurmstepd: error: *** JOB 19930531 ON orion-01-65 CANCELLED AT 2025-03-23T13:16:29 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/gdas_fcst_seg0.log: 0: slurmstepd: error: *** STEP 19930594.0 ON orion-03-11 CANCELLED AT 2025-03-23T13:34:00 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/gdas_fcst_seg0.log:slurmstepd: error: *** JOB 19930594 ON orion-03-11 CANCELLED AT 2025-03-23T13:34:00 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmaerosnowDA/COMROOT/C96C48_hybatmaerosnowDA/logs/2021122012/gdas_fcst_seg0.log.0:slurmstepd: error: *** JOB 19930519 ON orion-03-31 CANCELLED AT 2025-03-23T13:13:29 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/logs/2024022400/gdas_prepatmiodaobs.log.0:slurmstepd: error: *** JOB 19930625 ON orion-01-43 CANCELLED AT 2025-03-23T13:54:01 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C96C48_hybatmDA/COMROOT/C96C48_hybatmDA/logs/2021122018/gdas_fcst_seg0.log.0:slurmstepd: error: *** JOB 19930523 ON orion-16-12 CANCELLED AT 2025-03-23T13:13:29 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_hybAOWCDA/COMROOT/C48mx500_hybAOWCDA/logs/2021032418/enkfgdas_fcst_mem001.log.0:slurmstepd: error: *** JOB 19930525 ON orion-02-70 CANCELLED AT 2025-03-23T13:14:29 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_hybAOWCDA/COMROOT/C48mx500_hybAOWCDA/logs/2021032418/enkfgdas_fcst_mem002.log.0:slurmstepd: error: *** JOB 19930526 ON orion-18-12 CANCELLED AT 2025-03-23T13:14:29 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_hybAOWCDA/COMROOT/C48mx500_hybAOWCDA/logs/2021032500/gdas_marineanlinit.log.0:slurmstepd: error: *** JOB 19930693 ON orion-18-10 CANCELLED AT 2025-03-23T14:23:01 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_hybAOWCDA/COMROOT/C48mx500_hybAOWCDA/logs/2021032500/gdas_marinebmat.log.0:slurmstepd: error: *** JOB 19930655 ON orion-03-31 CANCELLED AT 2025-03-23T14:10:02 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_3DVarAOWCDA/COMROOT/C48mx500_3DVarAOWCDA/logs/2021032500/gdas_prepoceanobs.log.0:slurmstepd: error: *** JOB 19930542 ON orion-03-11 CANCELLED AT 2025-03-23T13:12:29 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_3DVarAOWCDA/COMROOT/C48mx500_3DVarAOWCDA/logs/2021032500/gdas_marineanlinit.log:slurmstepd: error: *** JOB 19930687 ON orion-03-31 CANCELLED AT 2025-03-23T14:20:32 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_3DVarAOWCDA/COMROOT/C48mx500_3DVarAOWCDA/logs/2021032500/gdas_marinebmat.log.0:slurmstepd: error: *** JOB 19930550 ON orion-01-05 CANCELLED AT 2025-03-23T13:34:00 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_3DVarAOWCDA/COMROOT/C48mx500_3DVarAOWCDA/logs/2021032500/gdas_marinebmat.log:slurmstepd: error: *** JOB 19930666 ON orion-18-10 CANCELLED AT 2025-03-23T14:10:02 DUE TO TIME LIMIT ***
gdas/test/gw-ci/C48mx500_3DVarAOWCDA/COMROOT/C48mx500_3DVarAOWCDA/logs/2021032500/gdas_prepoceanobs.log:slurmstepd: error: *** JOB 19930590 ON orion-01-43 CANCELLED AT 2025-03-23T13:23:00 DUE TO TIME LIMIT ***
gdas/test/atm/global-workflow/testrun/atmensanlinit-19930508.out:slurmstepd: error: *** JOB 19930508 ON orion-14-56 CANCELLED AT 2025-03-23T13:01:59 DUE TO TIME LIMIT ***
gdas/test/atm/global-workflow/testrun/atmensanlinit_split-19930551.out:slurmstepd: error: *** JOB 19930551 ON orion-14-56 CANCELLED AT 2025-03-23T13:14:29 DUE TO TIME LIMIT ***
gdas/test/atm/global-workflow/testrun/atmanlvar-19930541.out:slurmstepd: error: *** JOB 19930541 ON orion-15-33 CANCELLED AT 2025-03-23T13:12:29 DUE TO TIME LIMIT ***
gdas/test/atm/global-workflow/testrun/atmanlfinal-19930592.out:slurmstepd: error: *** JOB 19930592 ON orion-14-56 CANCELLED AT 2025-03-23T13:23:30 DUE TO TIME LIMIT ***
gdas/test/atm/global-workflow/testrun/atmensanlfinal-19930603.out:slurmstepd: error: *** JOB 19930603 ON orion-14-56 CANCELLED AT 2025-03-23T13:27:29 DUE TO TIME LIMIT ***
gdas/test/atm/global-workflow/testrun/atmanlinit-19930509.out:slurmstepd: error: *** JOB 19930509 ON orion-14-56 CANCELLED AT 2025-03-23T13:01:59 DUE TO TIME LIMIT ***

GDASApp ctests run in work2/noaa/stmp. It appears the /work2 fileset is experiencing IO issues.

A ticket with the Orion help desk will be opened to report this problem.

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor Author

Hercules and Orion /work2 performance*

The following tickets have been opened with the Hercules and Orion help desks reporting sluggish /work2 IO

  • Hercules: 2025032354000018
  • Orion: 2025032354000027

@emcbot
Copy link
Copy Markdown

emcbot commented Mar 23, 2025

Automated GW-GDASApp Testing Results:
Machine: orion

Start: Sun Mar 23 07:53:58 AM CDT 2025 on orion-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sun Mar 23 12:47:37 PM CDT 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                  *Failed*
Tests: Failed at Sun Mar 23 03:26:12 PM CDT 2025
Tests: 76% tests passed, 32 tests failed out of 136
	2042 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_fcst_202112201200 (Failed)
	2043 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_atmos_prod_202112201200 (Failed)
	2044 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_aeroanlgenb_202112201200 (Failed)
	2047 - test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_echgres_202112201200 (Failed)
	2049 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_prep_202112201800 (Failed)
	2050 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_anal_202112201800 (Failed)
	2051 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_aeroanlinit_202112201800 (Failed)
	2052 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_aeroanlvar_202112201800 (Failed)
	2053 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_aeroanlfinal_202112201800 (Failed)
	2054 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_snowanl_202112201800 (Failed)
	2055 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_sfcanl_202112201800 (Failed)
	2056 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_analcalc_202112201800 (Failed)
	2057 - test_gdasapp_C96C48_hybatmaerosnowDA_gdas_fcst_202112201800 (Failed)
	2061 - test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_ecmn_202112201800 (Failed)
	2063 - test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_esfc_202112201800 (Failed)
	2064 - test_gdasapp_C96C48_hybatmaerosnowDA_enkfgdas_fcst_202112201800 (Failed)
	2069 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_prepoceanobs_202103250000 (Failed)
	2070 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marinebmat_202103250000 (Failed)
	2071 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlinit_202103250000 (Failed)
	2072 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlvar_202103250000 (Failed)
	2073 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlchkpt_202103250000 (Failed)
	2074 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlfinal_202103250000 (Failed)
	2091 - test_gdasapp_atm_jjob_var_init (Failed)
	2093 - test_gdasapp_atm_jjob_var_inc (Failed)
	2094 - test_gdasapp_atm_jjob_var_final (Failed)
	2095 - test_gdasapp_atm_jjob_ens_init (Failed)
	2096 - test_gdasapp_atm_jjob_ens_letkf (Failed)
	2097 - test_gdasapp_atm_jjob_ens_init_split (Failed)
	2098 - test_gdasapp_atm_jjob_ens_obs (Failed)
	2099 - test_gdasapp_atm_jjob_ens_sol (Failed)
	2100 - test_gdasapp_atm_jjob_ens_inc (Failed)
	2101 - test_gdasapp_atm_jjob_ens_final (Failed)
Tests: see output at /work2/noaa/da/role-da/CI/orion/GDASApp/workflow/PR/1567/global-workflow/sorc/gdas.cd/build/log.ctest

@emcbot emcbot added orion-GW-RT-Failed Automated testing with global-workflow failed on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Mar 23, 2025
@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor Author

WCOSS2 g-w test

Install g-w develop at bf9a7a2 and feature/stable-nightly at 2bdc3c5 on Dogwood. Run g-w CI with the following results

/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_ATM_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Mar 23 2025 19:08:49    Mar 23 2025 20:15:27
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_3DVarAOWCDA_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Mar 23 2025 19:08:51    Mar 23 2025 19:25:28
202103250000        Done    Mar 23 2025 19:08:51    Mar 23 2025 21:10:30
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_hybAOWCDA_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Mar 23 2025 19:08:52    Mar 23 2025 19:25:31
202103250000        Done    Mar 23 2025 19:08:52    Mar 23 2025 20:30:33
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_S2SWA_gefs_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Mar 23 2025 19:08:54    Mar 23 2025 21:10:35
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_S2SW_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Mar 23 2025 19:08:56    Mar 23 2025 20:25:37
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96_atm3DVar_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Mar 23 2025 19:08:58    Mar 23 2025 19:25:43
202112210000        Done    Mar 23 2025 19:08:58    Mar 23 2025 21:35:35
202112210600        Done    Mar 23 2025 19:08:58    Mar 23 2025 21:20:36
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmaerosnowDA_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Mar 23 2025 19:09:00    Mar 23 2025 19:30:51
202112201800        Done    Mar 23 2025 19:09:00    Mar 23 2025 21:35:38
202112210000        Done    Mar 23 2025 19:09:00    Mar 23 2025 21:25:34
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmDA_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Mar 23 2025 19:09:02    Mar 23 2025 19:25:49
202112210000        Done    Mar 23 2025 19:09:02    Mar 23 2025 21:10:47
202112210600        Done    Mar 23 2025 19:09:02    Mar 23 2025 21:15:45
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_ufs_hybatmDA_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    Mar 23 2025 19:09:04    Mar 23 2025 19:25:53
202402240000        Done    Mar 23 2025 19:09:04    Mar 23 2025 22:05:40
202402240600        Done    Mar 23 2025 19:09:04    Mar 23 2025 21:55:40
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96mx100_S2S_stable-nightly
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
199405010000        Done    Mar 23 2025 19:09:06    Mar 23 2025 20:35:55

All g-w CI cases successfully run to completion on Dogwood.

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor Author

While GDASApp ctests fail on Hercules and Orion, these failures are due to ctest batch jobs running slow and exceeding the specified wall clock limit. All ctests pass on Hera. Tickets have been opened with the MSU help desk reporting the timeout behavior.

While it would be great to have GDASApp ctests pass on Hercules and Orion in addition to Hera, the Passed result on Hera and successful g-w CI on WCOSS2 (Dogwood) may be sufficient for approval of this PR.

We'll let the reviewers chime and and go from there.

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor Author

Manually rerun test_gdasapp_setup_atm_jjob_cycled_exp and test_gdasapp_atm_jjob suite of jobs on Hercules and Orion. Several of the jobs continue to fail with CANCELLED AT ... DUE TO TIME LIMIT. Something is not right with the MSU machines.

@RussTreadon-NOAA RussTreadon-NOAA merged commit 300b487 into develop Mar 24, 2025
@RussTreadon-NOAA RussTreadon-NOAA deleted the feature/stable-nightly branch March 24, 2025 13:23
@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor Author

Thank you @CoryMartin-NOAA and @DavidNew-NOAA . Updated hashes merged into develop.

jspsl pushed a commit that referenced this pull request Mar 26, 2025
danholdaway added a commit that referenced this pull request Mar 26, 2025
* develop:
  added original date (#1570)
  Update OCEAN_BASIN_FILE_ORION path for marine ctest (#1569)
  Fixing window_begin and end in yaml for redating ADT (#1565)
  Save LETKF mean analysis and mean increment for verification (#1566)
  Update JEDI hashes (20250324) (#1567)
  Removes obsolete json files from marine bufr2ioda (#1564)
  Add hybrid 4denvar as a phase3 validation configuration (#1552)
  Bugfix/ocn ecen (#1550)
  Bugfix SSS obs for LETKF (#1560)
  Bugfix ocean ensemble recentering (#1558)
  apply ci path patch only when running g-w ctests (#1554)
  Update JEDI hashes (20250317)  (#1531)
  add option to create_experiment.py in create_exp.sh  (#1546)
  Update jcb hashes and support initial IODA stats capability (#1547)
  Fix GW-CI CTest dependencies (#1534)
  Converts prepoceanobs to use JCB-style obs list and config files (#1527)
  JEDI-based ensemble recentering and analysis calculation (#1488)
  tools for phase3 validation (#1526)
  Update JEDI hashes (20250306) (#1520)
  New jcb-gdas # and insitu obs prep bug fix  (#1523)
  Marine in situ obs update (SSS, moorings, drifters) (#1511)
  Save marine background and analysis spread (#1517)
  Minor build and install updates and ctest cleanup (#1512)
  correct logic for upper loop limit in chem_diagb (#1516)
  Incorporate multi-cycle concatination and separation of variables in post-BUFR observations (#1504)
  Update JEDI hashes (20250225) (#1475)
  Add a depth dependent e-fold scale (#1494)
  Update jcb and da-utils hashes to support land DA. (#1509)
  Add options to inflate the ensemble while recentering (#1501)
  Add components to use jcb yamls for marine letkf job (#1486)
  Add GNSSRO bufr2ioda and json files (#1448)
  Removal of the marine verification tools (#1499)
  update logic to identify MSU machine (#1498)
  IODA Converter with In-Situ Observations Concatenation and ObsError Inflation (#1472)
  Add sea ice recentering (#1478)
  marine vrfy bugfix (#1483)
  add LD_LIBRARY_PATH patch for wcoss2 build (#1484)
  Bugfix for ensemble recenter + add a test (#1477)
  Add code changes for the implementation of the  low-resolution B (#1441)
  update spack-stack path in Hera modulefiles (#1471)
  fixed glider output filename (#1469)
  add gdas_waveinit to C48mx500_3DVarAOWCDA ctest suite (#1470)
  update jcb-gdas hash 20250128 (#1466)
  Time series of omb stats and html doc (#1463)
  Update marine vrfy HTML codes (#1464)
  Automate the verify tools to submit from start pdy to end pdy (#1458)
  Update an offline marine vrfy tool (#1447)
  Update JEDI hashes (20250127) (#1460)
DavidNew-NOAA pushed a commit that referenced this pull request Jan 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hera-GW-RT-Passed Automated testing with global-workflow successful on Hera hercules-GW-RT-Failed Automated testing with global-workflow failed on Hera orion-GW-RT-Failed Automated testing with global-workflow failed on Orion

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants