Update GDASApp to spack-stack/1.9.2 by RussTreadon-NOAA · Pull Request #1684 · NOAA-EMC/GDASApp

RussTreadon-NOAA · 2025-05-13T12:31:55Z

Description

This PR updates the GDASApp build to spack-stack/1.9.2

Companion PRs

none

Issues

Resolves #1283

Automated CI tests to run in Global Workflow

…ulefiles (#1283)

…ack191

RussTreadon-NOAA · 2025-05-13T12:36:08Z

20250513 update

This PR has, at present, one blocker. CRTM generates a segmentation fault when built with spack-stack/1.9.1. The problem is likely not spack-stack/1.9.1 but rather the updated Intel compilers used in the spack-stack/1.9.1 build. GDASApp issue #1671 documents this problem. If a large omp loop in CRTM_K_Matrix_Module.f90 is commented out, CRTM runs to completion using the new Intel compilers.

emcbot · 2025-06-24T15:56:49Z

Automated GW-GDASApp Testing Results:
Machine: hera

Start: Tue Jun 24 13:56:29 UTC 2025 on hfe10
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Tue Jun 24 14:35:08 UTC 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp -E C96C48_hybatmsnowDA|C96_gcafs_cycled
Tests:                                 *SUCCESS*
Tests: Completed at Tue Jun 24 15:56:47 UTC 2025
Tests: 100% tests passed, 0 tests failed out of 118

DavidHuber-NOAA · 2025-06-25T20:32:21Z

@RussTreadon-NOAA I am at the point of being able to test this branch in NOAA-EMC/global-workflow#3798 on C6. Do you know if there are any changes required to the workflow to make this branch work (outside of module updates)?

RussTreadon-NOAA · 2025-06-25T20:37:32Z

@DavidHuber-NOAA , Hera tests passed without any changes to g-w. Hopefully this holds true on Gaea C6.

Caveat: I have not extensively tested GDASApp on Gaea C6. This PR may not work on the first Gaea C6 g-w attempt.

RussTreadon-NOAA · 2025-06-27T15:08:48Z

Hera ctests

Install feature/spack-stack191 at f4cbf57 inside g-w develop at b60d23e. Run GDASApp ctests with the following results:

Test project /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/ss_191/sorc/gdas.cd/build
        Start 2068: test_gdasapp_C96C48_ufs_hybatmDA
        Start 2049: test_gdasapp_C96C48_hybatmDA
        Start 2092: test_gdasapp_C96_gcafs_cycled
        Start 2107: test_gdasapp_C96C48_hybatmsnowDA
        Start 2128: test_gdasapp_C48mx500_3DVarAOWCDA
        Start 2138: test_gdasapp_C48mx500_hybAOWCDA
        Start 2153: test_gdasapp_setup_atm_jjob_cycled_exp
        Start 1650: test_gdasapp_utils_incrhandler
        Start 1651: test_gdasapp_utils_incrhandler_8pes
        Start 1659: test_gdasapp_util_prepdata
        Start 1653: test_gdasapp_utils_hybridweights
        Start 1654: test_gdasapp_utils_setcorscales
  1/155 Test #2153: test_gdasapp_setup_atm_jjob_cycled_exp ...................................   Passed    1.24 sec
        Start 2158: test_gdasapp_atm_jjob_ens_init
  2/155 Test #1659: test_gdasapp_util_prepdata ...............................................   Passed    1.50 sec
        Start 2154: test_gdasapp_atm_jjob_var_init
  3/155 Test #2092: test_gdasapp_C96_gcafs_cycled ............................................   Passed    6.98 sec
        Start 2093: test_gdasapp_C96_gcafs_cycled_gcdas_stage_ic_202112201200
  4/155 Test #2128: test_gdasapp_C48mx500_3DVarAOWCDA ........................................   Passed    8.74 sec
        Start 2129: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_stage_ic_202103241800
  5/155 Test #2138: test_gdasapp_C48mx500_hybAOWCDA ..........................................   Passed    9.76 sec
        Start 2130: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_waveinit_202103241800
  6/155 Test #2049: test_gdasapp_C96C48_hybatmDA .............................................   Passed   10.46 sec
        Start 2050: test_gdasapp_C96C48_hybatmDA_gdas_stage_ic_202112201800
  7/155 Test #2068: test_gdasapp_C96C48_ufs_hybatmDA .........................................   Passed   10.68 sec
        Start 2069: test_gdasapp_C96C48_ufs_hybatmDA_gdas_stage_ic_202402231800
  8/155 Test #2107: test_gdasapp_C96C48_hybatmsnowDA .........................................   Passed   10.83 sec
        Start 2053: test_gdasapp_C96C48_hybatmDA_enkfgdas_stage_ic_202112201800
  9/155 Test #1654: test_gdasapp_utils_setcorscales ..........................................   Passed   16.45 sec
        Start 2072: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_stage_ic_202402231800
 10/155 Test #1650: test_gdasapp_utils_incrhandler ...........................................   Passed   16.51 sec
        Start 2108: test_gdasapp_C96C48_hybatmsnowDA_gdas_stage_ic_202112201200
 11/155 Test #1653: test_gdasapp_utils_hybridweights .........................................   Passed   16.81 sec
        Start 2111: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_stage_ic_202112201200
 12/155 Test #1651: test_gdasapp_utils_incrhandler_8pes ......................................   Passed   16.99 sec
        Start 2139: test_gdasapp_C48mx500_hybAOWCDA_gdas_stage_ic_202103241800
 13/155 Test #2093: test_gdasapp_C96_gcafs_cycled_gcdas_stage_ic_202112201200 ................   Passed   34.53 sec
        Start 2094: test_gdasapp_C96_gcafs_cycled_gcdas_fcst_202112201200
 14/155 Test #2129: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_stage_ic_202103241800 .............   Passed   34.40 sec
        Start 2141: test_gdasapp_C48mx500_hybAOWCDA_enkfgdas_stage_ic_202103241800
 15/155 Test #2158: test_gdasapp_atm_jjob_ens_init ...........................................   Passed   45.64 sec
        Start 2159: test_gdasapp_atm_jjob_ens_letkf
 16/155 Test #2154: test_gdasapp_atm_jjob_var_init ...........................................   Passed   45.60 sec
        Start 2097: test_gdasapp_C96_gcafs_cycled_gcdas_fetch_202112201800
 17/155 Test #2050: test_gdasapp_C96C48_hybatmDA_gdas_stage_ic_202112201800 ..................   Passed   37.03 sec
        Start 2051: test_gdasapp_C96C48_hybatmDA_gdas_fcst_202112201800
 18/155 Test #2139: test_gdasapp_C48mx500_hybAOWCDA_gdas_stage_ic_202103241800 ...............   Passed   36.11 sec
        Start 2140: test_gdasapp_C48mx500_hybAOWCDA_gdas_fcst_202103241800
 19/155 Test #2130: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_waveinit_202103241800 .............   Passed   48.44 sec
        Start 2131: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_fcst_202103241800
 20/155 Test #2053: test_gdasapp_C96C48_hybatmDA_enkfgdas_stage_ic_202112201800 ..............   Passed   49.24 sec
        Start 2054: test_gdasapp_C96C48_hybatmDA_enkfgdas_fcst_202112201800
 21/155 Test #2069: test_gdasapp_C96C48_ufs_hybatmDA_gdas_stage_ic_202402231800 ..............   Passed   50.89 sec
        Start 2070: test_gdasapp_C96C48_ufs_hybatmDA_gdas_fcst_202402231800
 22/155 Test #2072: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_stage_ic_202402231800 ..........   Passed   46.31 sec
        Start 2073: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_fcst_202402231800
 23/155 Test #2108: test_gdasapp_C96C48_hybatmsnowDA_gdas_stage_ic_202112201200 ..............   Passed   46.77 sec
        Start 2109: test_gdasapp_C96C48_hybatmsnowDA_gdas_fcst_202112201200
 24/155 Test #2111: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_stage_ic_202112201200 ..........   Passed   47.16 sec
        Start 2112: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_fcst_202112201200
 25/155 Test #2141: test_gdasapp_C48mx500_hybAOWCDA_enkfgdas_stage_ic_202103241800 ...........   Passed   45.19 sec
        Start 2142: test_gdasapp_C48mx500_hybAOWCDA_enkfgdas_fcst_202103241800
 26/155 Test #2097: test_gdasapp_C96_gcafs_cycled_gcdas_fetch_202112201800 ...................   Passed   71.95 sec
        Start 2105: test_gdasapp_C96_gcafs_cycled_gcdas_prep_emissions_202112201800
 27/155 Test #2105: test_gdasapp_C96_gcafs_cycled_gcdas_prep_emissions_202112201800 ..........   Passed   44.17 sec
        Start 2155: test_gdasapp_atm_jjob_var_run
 28/155 Test #2073: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_fcst_202402231800 ..............   Passed  293.97 sec
        Start 2074: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_epmn_202402231800
 29/155 Test #2054: test_gdasapp_C96C48_hybatmDA_enkfgdas_fcst_202112201800 ..................   Passed  310.95 sec
        Start 2056: test_gdasapp_C96C48_hybatmDA_enkfgdas_epmn_202112201800
 30/155 Test #2112: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_fcst_202112201200 ..............   Passed  307.62 sec
        Start 2114: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_epmn_202112201200
 31/155 Test #2140: test_gdasapp_C48mx500_hybAOWCDA_gdas_fcst_202103241800 ...................   Passed  355.75 sec
        Start 2143: test_gdasapp_C48mx500_hybAOWCDA_gdas_prepoceanobs_202103250000
 32/155 Test #2074: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_epmn_202402231800 ..............   Passed   63.94 sec
        Start 1652: test_gdasapp_utils_incrhandler_reproducibility
 33/155 Test #1652: test_gdasapp_utils_incrhandler_reproducibility ...........................   Passed    0.07 sec
        Start 1655: test_gdasapp_utils_diagb
 34/155 Test #2056: test_gdasapp_C96C48_hybatmDA_enkfgdas_epmn_202112201800 ..................   Passed   59.97 sec
        Start 1656: test_gdasapp_utils_enshandler
 35/155 Test #2114: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_epmn_202112201200 ..............   Passed   59.83 sec
        Start 1657: test_gdasapp_utils_socatofv3
 36/155 Test #2131: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_fcst_202103241800 .................   Passed  373.42 sec
        Start 2132: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_prepoceanobs_202103250000
 37/155 Test #1655: test_gdasapp_utils_diagb .................................................   Passed   11.77 sec
        Start 2133: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marinebmat_202103250000
 38/155 Test #1656: test_gdasapp_utils_enshandler ............................................   Passed    6.88 sec
        Start 1658: test_gdasapp_util_coding_norms
 39/155 Test #1657: test_gdasapp_utils_socatofv3 .............................................   Passed    8.78 sec
        Start 1660: test_gdasapp_util_rads2ioda
 40/155 Test #1658: test_gdasapp_util_coding_norms ...........................................   Passed    3.44 sec
        Start 1661: test_gdasapp_util_ghrsst2ioda
 41/155 Test #1660: test_gdasapp_util_rads2ioda ..............................................   Passed    1.68 sec
        Start 1662: test_gdasapp_util_rtofstmp
 42/155 Test #1661: test_gdasapp_util_ghrsst2ioda ............................................   Passed    0.59 sec
        Start 1663: test_gdasapp_util_rtofssal
 43/155 Test #1663: test_gdasapp_util_rtofssal ...............................................   Passed    0.92 sec
        Start 1664: test_gdasapp_util_smap2ioda
 44/155 Test #1662: test_gdasapp_util_rtofstmp ...............................................   Passed    1.05 sec
        Start 1665: test_gdasapp_util_smos2ioda
 45/155 Test #1664: test_gdasapp_util_smap2ioda ..............................................   Passed    0.46 sec
        Start 1666: test_gdasapp_util_viirsaod2ioda
 46/155 Test #1665: test_gdasapp_util_smos2ioda ..............................................   Passed    0.36 sec
        Start 1667: test_gdasapp_util_icecabi2ioda
 47/155 Test #1666: test_gdasapp_util_viirsaod2ioda ..........................................   Passed    0.09 sec
        Start 1668: test_gdasapp_util_icecamsr2ioda
 48/155 Test #1667: test_gdasapp_util_icecabi2ioda ...........................................   Passed    0.10 sec
        Start 1669: test_gdasapp_util_icecmirs2ioda
 49/155 Test #1668: test_gdasapp_util_icecamsr2ioda ..........................................   Passed    0.41 sec
        Start 1670: test_gdasapp_util_icecjpssrr2ioda
 50/155 Test #1669: test_gdasapp_util_icecmirs2ioda ..........................................   Passed    0.41 sec
        Start 1671: test_gdasapp_util_insituall2ioda
 51/155 Test #1670: test_gdasapp_util_icecjpssrr2ioda ........................................   Passed    0.09 sec
        Start 2043: test_gdasapp_check_python_norms
 52/155 Test #1671: test_gdasapp_util_insituall2ioda .........................................   Passed    0.09 sec
        Start 2044: test_gdasapp_check_yaml_keys
 53/155 Test #2044: test_gdasapp_check_yaml_keys .............................................   Passed    0.13 sec
        Start 2045: test_gdasapp_snow_create_ens
 54/155 Test #2045: test_gdasapp_snow_create_ens .............................................   Passed    1.02 sec
        Start 2046: test_gdasapp_snow_imsproc
 55/155 Test #2043: test_gdasapp_check_python_norms ..........................................   Passed    3.09 sec
        Start 2047: test_gdasapp_snow_apply_jediincr
 56/155 Test #2046: test_gdasapp_snow_imsproc ................................................   Passed    4.55 sec
        Start 2048: test_gdasapp_convert_bufr_adpsfc_snow
 57/155 Test #2048: test_gdasapp_convert_bufr_adpsfc_snow ....................................   Passed    3.50 sec
        Start 2151: test_gdasapp_convert_bufr_adpsfc
 58/155 Test #2047: test_gdasapp_snow_apply_jediincr .........................................   Passed    6.36 sec
        Start 2152: test_gdasapp_convert_gsi_satbias
 59/155 Test #2151: test_gdasapp_convert_bufr_adpsfc .........................................   Passed    3.65 sec
        Start 2165: test_gdasapp_aero_gen_3dvar_yaml
 60/155 Test #2165: test_gdasapp_aero_gen_3dvar_yaml .........................................   Passed    0.37 sec
        Start 2166: test_gdasapp_bufr2ioda_insitu_profile_argo
 61/155 Test #2152: test_gdasapp_convert_gsi_satbias .........................................   Passed    4.90 sec
        Start 2167: test_gdasapp_bufr2ioda_insitu_profile_bathy
 62/155 Test #2167: test_gdasapp_bufr2ioda_insitu_profile_bathy ..............................   Passed    1.16 sec
        Start 2168: test_gdasapp_bufr2ioda_insitu_profile_glider
 63/155 Test #2168: test_gdasapp_bufr2ioda_insitu_profile_glider .............................   Passed    2.02 sec
        Start 2169: test_gdasapp_bufr2ioda_insitu_profile_tesac
 64/155 Test #2166: test_gdasapp_bufr2ioda_insitu_profile_argo ...............................   Passed    7.19 sec
        Start 2170: test_gdasapp_bufr2ioda_insitu_profile_tropical
 65/155 Test #2170: test_gdasapp_bufr2ioda_insitu_profile_tropical ...........................   Passed    2.23 sec
        Start 2171: test_gdasapp_bufr2ioda_insitu_profile_xbtctd
 66/155 Test #2171: test_gdasapp_bufr2ioda_insitu_profile_xbtctd .............................   Passed    2.24 sec
        Start 2172: test_gdasapp_bufr2ioda_insitu_surface_drifter
 67/155 Test #2172: test_gdasapp_bufr2ioda_insitu_surface_drifter ............................   Passed    1.53 sec
        Start 2173: test_gdasapp_bufr2ioda_insitu_surface_trkob
 68/155 Test #2173: test_gdasapp_bufr2ioda_insitu_surface_trkob ..............................   Passed    1.70 sec
        Start 2174: test_gdasapp_bufr2ioda_insitu_surface_dbuoyb
 69/155 Test #2169: test_gdasapp_bufr2ioda_insitu_profile_tesac ..............................   Passed   11.55 sec
        Start 2175: test_gdasapp_bufr2ioda_insitu_surface_dbuoyb_drifter
 70/155 Test #2174: test_gdasapp_bufr2ioda_insitu_surface_dbuoyb .............................   Passed    2.32 sec
 71/155 Test #2175: test_gdasapp_bufr2ioda_insitu_surface_dbuoyb_drifter .....................   Passed    1.53 sec
 72/155 Test #2142: test_gdasapp_C48mx500_hybAOWCDA_enkfgdas_fcst_202103241800 ...............   Passed  388.52 sec
        Start 2144: test_gdasapp_C48mx500_hybAOWCDA_gdas_marinebmat_202103250000
 73/155 Test #2133: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marinebmat_202103250000 ...........   Passed   75.50 sec
 74/155 Test #2051: test_gdasapp_C96C48_hybatmDA_gdas_fcst_202112201800 ......................   Passed  487.00 sec
        Start 2052: test_gdasapp_C96C48_hybatmDA_gdas_atmos_prod_202112201800
        Start 2055: test_gdasapp_C96C48_hybatmDA_enkfgdas_echgres_202112201800
 75/155 Test #2109: test_gdasapp_C96C48_hybatmsnowDA_gdas_fcst_202112201200 ..................   Passed  471.21 sec
        Start 2110: test_gdasapp_C96C48_hybatmsnowDA_gdas_atmos_prod_202112201200
        Start 2113: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_echgres_202112201200
 76/155 Test #2070: test_gdasapp_C96C48_ufs_hybatmDA_gdas_fcst_202402231800 ..................   Passed  477.08 sec
        Start 2071: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmos_prod_202402231800
 77/155 Test #2144: test_gdasapp_C48mx500_hybAOWCDA_gdas_marinebmat_202103250000 .............   Passed   87.86 sec
 78/155 Test #2113: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_echgres_202112201200 ...........   Passed   46.75 sec
 79/155 Test #2055: test_gdasapp_C96C48_hybatmDA_enkfgdas_echgres_202112201800 ...............   Passed   47.06 sec
 80/155 Test #2159: test_gdasapp_atm_jjob_ens_letkf ..........................................   Passed  554.28 sec
        Start 2160: test_gdasapp_atm_jjob_ens_init_split
 81/155 Test #2160: test_gdasapp_atm_jjob_ens_init_split .....................................   Passed   45.65 sec
        Start 2161: test_gdasapp_atm_jjob_ens_obs
 82/155 Test #2052: test_gdasapp_C96C48_hybatmDA_gdas_atmos_prod_202112201800 ................   Passed  137.55 sec
        Start 2057: test_gdasapp_C96C48_hybatmDA_gdas_prep_202112210000
 83/155 Test #2110: test_gdasapp_C96C48_hybatmsnowDA_gdas_atmos_prod_202112201200 ............   Passed  137.68 sec
        Start 2115: test_gdasapp_C96C48_hybatmsnowDA_gdas_prep_202112201800
 84/155 Test #2071: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmos_prod_202402231800 ............   Passed  137.14 sec
        Start 2075: test_gdasapp_C96C48_ufs_hybatmDA_gdas_prep_202402240000
 85/155 Test #2155: test_gdasapp_atm_jjob_var_run ............................................   Passed  522.24 sec
        Start 2156: test_gdasapp_atm_jjob_var_inc
 86/155 Test #2161: test_gdasapp_atm_jjob_ens_obs ............................................   Passed   42.17 sec
        Start 2162: test_gdasapp_atm_jjob_ens_sol
 87/155 Test #2143: test_gdasapp_C48mx500_hybAOWCDA_gdas_prepoceanobs_202103250000 ...........   Passed  281.78 sec
        Start 2146: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlinit_202103250000
        Start 2145: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlletkf_202103250000
 88/155 Test #2132: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_prepoceanobs_202103250000 .........   Passed  261.17 sec
        Start 2134: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlinit_202103250000
 89/155 Test #2156: test_gdasapp_atm_jjob_var_inc ............................................   Passed   42.15 sec
        Start 2157: test_gdasapp_atm_jjob_var_final
 90/155 Test #2162: test_gdasapp_atm_jjob_ens_sol ............................................   Passed   42.17 sec
        Start 2163: test_gdasapp_atm_jjob_ens_inc
 91/155 Test #2134: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlinit_202103250000 ........   Passed   61.26 sec
        Start 2135: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlvar_202103250000
 92/155 Test #2146: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlinit_202103250000 ..........   Passed   69.26 sec
        Start 2147: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlvar_202103250000
 93/155 Test #2157: test_gdasapp_atm_jjob_var_final ..........................................   Passed   42.15 sec
 94/155 Test #2163: test_gdasapp_atm_jjob_ens_inc ............................................   Passed   42.15 sec
        Start 2164: test_gdasapp_atm_jjob_ens_final
 95/155 Test #2164: test_gdasapp_atm_jjob_ens_final ..........................................   Passed   42.15 sec
 96/155 Test #2145: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlletkf_202103250000 .........   Passed  140.03 sec
 97/155 Test #2135: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlvar_202103250000 .........   Passed   77.23 sec
        Start 2136: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlchkpt_202103250000
 98/155 Test #2147: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlvar_202103250000 ...........   Passed   73.74 sec
        Start 2149: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlchkpt_202103250000
        Start 2148: test_gdasapp_C48mx500_hybAOWCDA_gdas_ocnanalecen_202103250000
 99/155 Test #2148: test_gdasapp_C48mx500_hybAOWCDA_gdas_ocnanalecen_202103250000 ............   Passed   68.71 sec
100/155 Test #2149: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlchkpt_202103250000 .........   Passed   70.24 sec
        Start 2150: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlfinal_202103250000
101/155 Test #2136: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlchkpt_202103250000 .......   Passed   72.89 sec
        Start 2137: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlfinal_202103250000
102/155 Test #2094: test_gdasapp_C96_gcafs_cycled_gcdas_fcst_202112201200 ....................   Passed  871.22 sec
        Start 2095: test_gdasapp_C96_gcafs_cycled_gcdas_atmos_prod_202112201200
        Start 2096: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlgenb_202112201200
        Start 2098: test_gdasapp_C96_gcafs_cycled_gcdas_offlineanl_202112201800
103/155 Test #2115: test_gdasapp_C96C48_hybatmsnowDA_gdas_prep_202112201800 ..................   Passed  267.19 sec
        Start 2116: test_gdasapp_C96C48_hybatmsnowDA_gdas_anal_202112201800
        Start 2121: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_eobs_202112201800
        Start 2117: test_gdasapp_C96C48_hybatmsnowDA_gdas_snowanl_202112201800
104/155 Test #2150: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlfinal_202103250000 .........   Passed   59.36 sec
105/155 Test #2137: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlfinal_202103250000 .......   Passed   61.52 sec
106/155 Test #2057: test_gdasapp_C96C48_hybatmDA_gdas_prep_202112210000 ......................   Passed  308.79 sec
        Start 2062: test_gdasapp_C96C48_hybatmDA_enkfgdas_eobs_202112210000
        Start 2058: test_gdasapp_C96C48_hybatmDA_gdas_anal_202112210000
107/155 Test #2096: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlgenb_202112201200 .............   Passed  120.79 sec
108/155 Test #2095: test_gdasapp_C96_gcafs_cycled_gcdas_atmos_prod_202112201200 ..............   Passed  120.80 sec
        Start 2104: test_gdasapp_C96_gcafs_cycled_gcdas_prepobsaero_202112201800
109/155 Test #2117: test_gdasapp_C96C48_hybatmsnowDA_gdas_snowanl_202112201800 ...............   Passed  109.64 sec
        Start 2125: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_esnowanl_202112201800
110/155 Test #2104: test_gdasapp_C96_gcafs_cycled_gcdas_prepobsaero_202112201800 .............   Passed   73.79 sec
        Start 2101: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlinit_202112201800
111/155 Test #2075: test_gdasapp_C96C48_ufs_hybatmDA_gdas_prep_202402240000 ..................   Passed  447.04 sec
        Start 2076: test_gdasapp_C96C48_ufs_hybatmDA_gdas_prepatmiodaobs_202402240000
112/155 Test #2125: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_esnowanl_202112201800 ..........   Passed  100.21 sec
113/155 Test #2098: test_gdasapp_C96_gcafs_cycled_gcdas_offlineanl_202112201800 ..............   Passed  253.10 sec
        Start 2099: test_gdasapp_C96_gcafs_cycled_gcdas_sfcanl_202112201800
114/155 Test #2099: test_gdasapp_C96_gcafs_cycled_gcdas_sfcanl_202112201800 ..................   Passed   62.59 sec
115/155 Test #2101: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlinit_202112201800 .............   Passed  151.61 sec
        Start 2102: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlvar_202112201800
116/155 Test #2121: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_eobs_202112201800 ..............   Passed  459.75 sec
        Start 2122: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_ediag_202112201800
117/155 Test #2062: test_gdasapp_C96C48_hybatmDA_enkfgdas_eobs_202112210000 ..................   Passed  477.09 sec
        Start 2063: test_gdasapp_C96C48_hybatmDA_enkfgdas_ediag_202112210000
118/155 Test #2076: test_gdasapp_C96C48_ufs_hybatmDA_gdas_prepatmiodaobs_202402240000 ........   Passed  388.88 sec
        Start 2084: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlinit_202402240000
        Start 2077: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlinit_202402240000
119/155 Test #2122: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_ediag_202112201800 .............   Passed  179.13 sec
        Start 2123: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_eupd_202112201800
120/155 Test #2077: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlinit_202402240000 ............   Passed   73.71 sec
        Start 2078: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlvar_202402240000
121/155 Test #2084: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlinit_202402240000 .....   Passed   74.53 sec
        Start 2085: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlobs_202402240000
122/155 Test #2063: test_gdasapp_C96C48_hybatmDA_enkfgdas_ediag_202112210000 .................   Passed  177.07 sec
        Start 2064: test_gdasapp_C96C48_hybatmDA_enkfgdas_eupd_202112210000
123/155 Test #2123: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_eupd_202112201800 ..............   Passed  163.48 sec
124/155 Test #2102: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlvar_202112201800 ..............***Failed  526.94 sec
        Start 2103: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlfinal_202112201800
125/155 Test #2064: test_gdasapp_C96C48_hybatmDA_enkfgdas_eupd_202112210000 ..................   Passed  183.23 sec
126/155 Test #2058: test_gdasapp_C96C48_hybatmDA_gdas_anal_202112210000 ......................   Passed  839.16 sec
        Start 2059: test_gdasapp_C96C48_hybatmDA_gdas_sfcanl_202112210000
127/155 Test #2103: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlfinal_202112201800 ............***Failed   59.60 sec
        Start 2106: test_gdasapp_C96_gcafs_cycled_gcdas_fcst_202112201800
128/155 Test #2116: test_gdasapp_C96C48_hybatmsnowDA_gdas_anal_202112201800 ..................   Passed  964.10 sec
        Start 2118: test_gdasapp_C96C48_hybatmsnowDA_gdas_sfcanl_202112201800
129/155 Test #2059: test_gdasapp_C96C48_hybatmDA_gdas_sfcanl_202112210000 ....................   Passed  102.10 sec
        Start 2060: test_gdasapp_C96C48_hybatmDA_gdas_analcalc_202112210000
        Start 2061: test_gdasapp_C96C48_hybatmDA_gdas_fcst_202112210000
130/155 Test #2118: test_gdasapp_C96C48_hybatmsnowDA_gdas_sfcanl_202112201800 ................   Passed   78.58 sec
        Start 2119: test_gdasapp_C96C48_hybatmsnowDA_gdas_analcalc_202112201800
        Start 2120: test_gdasapp_C96C48_hybatmsnowDA_gdas_fcst_202112201800
131/155 Test #2060: test_gdasapp_C96C48_hybatmDA_gdas_analcalc_202112210000 ..................   Passed   87.41 sec
        Start 2065: test_gdasapp_C96C48_hybatmDA_enkfgdas_ecmn_202112210000
        Start 2066: test_gdasapp_C96C48_hybatmDA_enkfgdas_esfc_202112210000
132/155 Test #2119: test_gdasapp_C96C48_hybatmsnowDA_gdas_analcalc_202112201800 ..............   Passed   87.95 sec
        Start 2124: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_ecmn_202112201800
        Start 2126: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_esfc_202112201800
133/155 Test #2065: test_gdasapp_C96C48_hybatmDA_enkfgdas_ecmn_202112210000 ..................   Passed   74.96 sec
134/155 Test #2085: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlobs_202402240000 ......   Passed  521.27 sec
        Start 2086: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlsol_202402240000
135/155 Test #2124: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_ecmn_202112201800 ..............   Passed   72.01 sec
136/155 Test #2066: test_gdasapp_C96C48_hybatmDA_enkfgdas_esfc_202112210000 ..................   Passed  167.82 sec
        Start 2067: test_gdasapp_C96C48_hybatmDA_enkfgdas_fcst_202112210000
137/155 Test #2086: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlsol_202402240000 ......   Passed   99.36 sec
        Start 2087: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlfv3inc_202402240000
138/155 Test #2126: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_esfc_202112201800 ..............   Passed  138.47 sec
        Start 2127: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_fcst_202112201800
139/155 Test #2087: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlfv3inc_202402240000 ...   Passed   58.53 sec
        Start 2088: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlfinal_202402240000
140/155 Test #2078: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlvar_202402240000 .............   Passed  707.49 sec
        Start 2079: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlfv3inc_202402240000
141/155 Test #2079: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlfv3inc_202402240000 ..........   Passed   60.42 sec
        Start 2080: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlfinal_202402240000
142/155 Test #2061: test_gdasapp_C96C48_hybatmDA_gdas_fcst_202112210000 ......................   Passed  572.93 sec
143/155 Test #2080: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlfinal_202402240000 ...........   Passed  151.47 sec
        Start 2081: test_gdasapp_C96C48_ufs_hybatmDA_gdas_sfcanl_202402240000
        Start 2082: test_gdasapp_C96C48_ufs_hybatmDA_gdas_analcalc_fv3jedi_202402240000
144/155 Test #2088: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlfinal_202402240000 ....   Passed  245.74 sec
        Start 2089: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_ecen_fv3jedi_202402240000
        Start 2090: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_esfc_202402240000
145/155 Test #2106: test_gdasapp_C96_gcafs_cycled_gcdas_fcst_202112201800 ....................   Passed  674.55 sec
        Start 2100: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlgenb_202112201800
146/155 Test #2067: test_gdasapp_C96C48_hybatmDA_enkfgdas_fcst_202112210000 ..................   Passed  358.93 sec
147/155 Test #2081: test_gdasapp_C96C48_ufs_hybatmDA_gdas_sfcanl_202402240000 ................   Passed   56.36 sec
        Start 2083: test_gdasapp_C96C48_ufs_hybatmDA_gdas_fcst_202402240000
148/155 Test #2120: test_gdasapp_C96C48_hybatmsnowDA_gdas_fcst_202112201800 ..................   Passed  581.76 sec
149/155 Test #2127: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_fcst_202112201800 ..............   Passed  356.15 sec
150/155 Test #2082: test_gdasapp_C96C48_ufs_hybatmDA_gdas_analcalc_fv3jedi_202402240000 ......   Passed   73.24 sec
151/155 Test #2089: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_ecen_fv3jedi_202402240000 ......   Passed   81.51 sec
152/155 Test #2100: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlgenb_202112201800 .............   Passed  115.04 sec
153/155 Test #2090: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_esfc_202402240000 ..............   Passed  141.92 sec
        Start 2091: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_fcst_202402240000
154/155 Test #2091: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_fcst_202402240000 ..............   Passed  298.12 sec
155/155 Test #2083: test_gdasapp_C96C48_ufs_hybatmDA_gdas_fcst_202402240000 ..................   Passed  470.13 sec

99% tests passed, 2 tests failed out of 155

Label Time Summary:
gdas-utils    =  11.20 sec*proc (14 tests)
gdasapp       = 21620.83 sec*proc (114 tests)
script        = 21632.03 sec*proc (128 tests)

Total Test time (real) = 3031.99 sec

The following tests FAILED:
        2102 - test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlvar_202112201800 (Failed)
        2103 - test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlfinal_202112201800 (Failed)

test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlvar_202112201800 failed due to an OOM error. The job ran 96 with ppn=40. A check of the log file shows per task memory usage of 2.56 Gb. 40 x 2.56 = 102.4 Gb. Hera compute nodes have 96 Gb of memory. Reconfigure job to run ppn=20 with a memory request of 96Gb per node. Job ran to completion. Others have reported that applications built with spack-stack/1.9.2 use more memory.

@CoryMartin-NOAA and @andytangborn : When we update GDASApp to spack-stack/1.9.2 we need to modify g-w dev/parm/config/gcafs/config.resources.HERA (actually a link to dev/parm/config/gfs/config.resources.HERA)

@@ -20,6 +20,11 @@ case ${step} in
     export tasks_per_node=12
     ;;

+  "aeroanlvar")
+    export tasks_per_node=20
+    export memory="96GB"
+    ;;
+
   "eupd")
     case "${CASE}" in
       "C1152" | "C768")

Reran test_gdasapp_C96_gcafs_cycled with a locally modified config.resources.Hera file. All jobs Passed

Test project /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/ss_191/sorc/gdas.cd/build
      Start 2092: test_gdasapp_C96_gcafs_cycled
 1/15 Test #2092: test_gdasapp_C96_gcafs_cycled .....................................   Passed    5.37 sec
      Start 2093: test_gdasapp_C96_gcafs_cycled_gcdas_stage_ic_202112201200
      Start 2097: test_gdasapp_C96_gcafs_cycled_gcdas_fetch_202112201800
      Start 2105: test_gdasapp_C96_gcafs_cycled_gcdas_prep_emissions_202112201800
 2/15 Test #2093: test_gdasapp_C96_gcafs_cycled_gcdas_stage_ic_202112201200 .........   Passed   38.81 sec
      Start 2094: test_gdasapp_C96_gcafs_cycled_gcdas_fcst_202112201200
 3/15 Test #2105: test_gdasapp_C96_gcafs_cycled_gcdas_prep_emissions_202112201800 ...   Passed   50.24 sec
 4/15 Test #2097: test_gdasapp_C96_gcafs_cycled_gcdas_fetch_202112201800 ............   Passed   78.22 sec
 5/15 Test #2094: test_gdasapp_C96_gcafs_cycled_gcdas_fcst_202112201200 .............   Passed  875.33 sec
      Start 2095: test_gdasapp_C96_gcafs_cycled_gcdas_atmos_prod_202112201200
      Start 2096: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlgenb_202112201200
      Start 2098: test_gdasapp_C96_gcafs_cycled_gcdas_offlineanl_202112201800
 6/15 Test #2095: test_gdasapp_C96_gcafs_cycled_gcdas_atmos_prod_202112201200 .......   Passed   97.53 sec
      Start 2104: test_gdasapp_C96_gcafs_cycled_gcdas_prepobsaero_202112201800
 7/15 Test #2096: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlgenb_202112201200 ......   Passed  111.60 sec
 8/15 Test #2104: test_gdasapp_C96_gcafs_cycled_gcdas_prepobsaero_202112201800 ......   Passed   57.97 sec
      Start 2101: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlinit_202112201800
 9/15 Test #2098: test_gdasapp_C96_gcafs_cycled_gcdas_offlineanl_202112201800 .......   Passed  254.97 sec
      Start 2099: test_gdasapp_C96_gcafs_cycled_gcdas_sfcanl_202112201800
10/15 Test #2101: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlinit_202112201800 ......   Passed  154.81 sec
      Start 2102: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlvar_202112201800
11/15 Test #2099: test_gdasapp_C96_gcafs_cycled_gcdas_sfcanl_202112201800 ...........   Passed   61.34 sec
12/15 Test #2102: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlvar_202112201800 .......   Passed  274.72 sec
      Start 2103: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlfinal_202112201800
13/15 Test #2103: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlfinal_202112201800 .....   Passed   73.63 sec
      Start 2106: test_gdasapp_C96_gcafs_cycled_gcdas_fcst_202112201800
14/15 Test #2106: test_gdasapp_C96_gcafs_cycled_gcdas_fcst_202112201800 .............   Passed  690.87 sec
      Start 2100: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlgenb_202112201800
15/15 Test #2100: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlgenb_202112201800 ......   Passed  103.54 sec

100% tests passed, 0 tests failed out of 15

Label Time Summary:
gdasapp    = 2928.96 sec*proc (15 tests)
script     = 2928.96 sec*proc (15 tests)

Total Test time (real) = 2373.34 sec

RussTreadon-NOAA · 2025-06-27T16:29:48Z

Orion and Hercules ctests

Install feature/spack-stack191 at f4cbf57 inside g-w develop at b60d23e. Run GDASApp ctests with the following results on Orion:

Test project /work2/noaa/da/rtreadon/git/global-workflow/ss_191/sorc/gdas.cd/build
        Start 2060: test_gdasapp_C96C48_ufs_hybatmDA
        Start 2041: test_gdasapp_C96C48_hybatmDA
        Start 2084: test_gdasapp_C96_gcafs_cycled
        Start 2099: test_gdasapp_C96C48_hybatmsnowDA
        Start 2120: test_gdasapp_C48mx500_3DVarAOWCDA
        Start 2130: test_gdasapp_C48mx500_hybAOWCDA
        Start 2145: test_gdasapp_setup_atm_jjob_cycled_exp
        Start 1651: test_gdasapp_util_prepdata
        Start 1650: test_gdasapp_util_coding_norms
        Start 1654: test_gdasapp_util_rtofstmp
        Start 1655: test_gdasapp_util_rtofssal
        Start 2035: test_gdasapp_check_python_norms
  1/147 Test #1655: test_gdasapp_util_rtofssal ...............................................   Passed    1.28 sec
        Start 2036: test_gdasapp_check_yaml_keys
  2/147 Test #2145: test_gdasapp_setup_atm_jjob_cycled_exp ...................................   Passed    1.33 sec
        Start 2150: test_gdasapp_atm_jjob_ens_init
  3/147 Test #1654: test_gdasapp_util_rtofstmp ...............................................   Passed    1.43 sec
        Start 2146: test_gdasapp_atm_jjob_var_init
  4/147 Test #2036: test_gdasapp_check_yaml_keys .............................................   Passed    0.21 sec
        Start 2037: test_gdasapp_snow_create_ens
  5/147 Test #1651: test_gdasapp_util_prepdata ...............................................   Passed    1.82 sec
        Start 1652: test_gdasapp_util_rads2ioda
  6/147 Test #1652: test_gdasapp_util_rads2ioda ..............................................   Passed    0.12 sec
        Start 1653: test_gdasapp_util_ghrsst2ioda
  7/147 Test #1653: test_gdasapp_util_ghrsst2ioda ............................................   Passed    0.12 sec
        Start 1656: test_gdasapp_util_smap2ioda
  8/147 Test #1656: test_gdasapp_util_smap2ioda ..............................................   Passed    0.11 sec
        Start 1657: test_gdasapp_util_smos2ioda
  9/147 Test #1657: test_gdasapp_util_smos2ioda ..............................................   Passed    0.11 sec
        Start 1658: test_gdasapp_util_viirsaod2ioda
 10/147 Test #1658: test_gdasapp_util_viirsaod2ioda ..........................................   Passed    0.11 sec
        Start 1659: test_gdasapp_util_icecabi2ioda
 11/147 Test #1659: test_gdasapp_util_icecabi2ioda ...........................................   Passed    0.14 sec
        Start 1660: test_gdasapp_util_icecamsr2ioda
 12/147 Test #1660: test_gdasapp_util_icecamsr2ioda ..........................................   Passed    0.13 sec
        Start 1661: test_gdasapp_util_icecmirs2ioda
 13/147 Test #1661: test_gdasapp_util_icecmirs2ioda ..........................................   Passed    0.16 sec
        Start 1662: test_gdasapp_util_icecjpssrr2ioda
 14/147 Test #1662: test_gdasapp_util_icecjpssrr2ioda ........................................   Passed    0.15 sec
        Start 1663: test_gdasapp_util_insituall2ioda
 15/147 Test #1663: test_gdasapp_util_insituall2ioda .........................................   Passed    0.13 sec
        Start 2038: test_gdasapp_snow_imsproc
 16/147 Test #2037: test_gdasapp_snow_create_ens .............................................   Passed    1.81 sec
        Start 2039: test_gdasapp_snow_apply_jediincr
 17/147 Test #2039: test_gdasapp_snow_apply_jediincr .........................................   Passed    2.67 sec
        Start 2040: test_gdasapp_convert_bufr_adpsfc_snow
 18/147 Test #1650: test_gdasapp_util_coding_norms ...........................................   Passed    6.82 sec
        Start 2143: test_gdasapp_convert_bufr_adpsfc
 19/147 Test #2038: test_gdasapp_snow_imsproc ................................................   Passed    3.75 sec
        Start 2144: test_gdasapp_convert_gsi_satbias
 20/147 Test #2035: test_gdasapp_check_python_norms ..........................................   Passed    9.06 sec
        Start 2157: test_gdasapp_aero_gen_3dvar_yaml
 21/147 Test #2157: test_gdasapp_aero_gen_3dvar_yaml .........................................   Passed    0.66 sec
        Start 2158: test_gdasapp_bufr2ioda_insitu_profile_argo
 22/147 Test #2040: test_gdasapp_convert_bufr_adpsfc_snow ....................................   Passed    7.28 sec
        Start 2159: test_gdasapp_bufr2ioda_insitu_profile_bathy
 23/147 Test #2143: test_gdasapp_convert_bufr_adpsfc .........................................   Passed    8.18 sec
        Start 2160: test_gdasapp_bufr2ioda_insitu_profile_glider
 24/147 Test #2159: test_gdasapp_bufr2ioda_insitu_profile_bathy ..............................   Passed    1.82 sec
        Start 2161: test_gdasapp_bufr2ioda_insitu_profile_tesac
 25/147 Test #2144: test_gdasapp_convert_gsi_satbias .........................................   Passed   10.33 sec
        Start 2162: test_gdasapp_bufr2ioda_insitu_profile_tropical
 26/147 Test #2160: test_gdasapp_bufr2ioda_insitu_profile_glider .............................   Passed    4.24 sec
        Start 2163: test_gdasapp_bufr2ioda_insitu_profile_xbtctd
 27/147 Test #2162: test_gdasapp_bufr2ioda_insitu_profile_tropical ...........................   Passed    2.30 sec
        Start 2164: test_gdasapp_bufr2ioda_insitu_surface_drifter
 28/147 Test #2084: test_gdasapp_C96_gcafs_cycled ............................................   Passed   19.90 sec
        Start 2085: test_gdasapp_C96_gcafs_cycled_gcdas_stage_ic_202112201200
 29/147 Test #2163: test_gdasapp_bufr2ioda_insitu_profile_xbtctd .............................   Passed    1.51 sec
        Start 2089: test_gdasapp_C96_gcafs_cycled_gcdas_fetch_202112201800
 30/147 Test #2164: test_gdasapp_bufr2ioda_insitu_surface_drifter ............................   Passed    1.91 sec
        Start 2097: test_gdasapp_C96_gcafs_cycled_gcdas_prep_emissions_202112201800
 31/147 Test #2158: test_gdasapp_bufr2ioda_insitu_profile_argo ...............................   Passed   12.81 sec
        Start 2165: test_gdasapp_bufr2ioda_insitu_surface_trkob
 32/147 Test #2165: test_gdasapp_bufr2ioda_insitu_surface_trkob ..............................   Passed    1.63 sec
        Start 2166: test_gdasapp_bufr2ioda_insitu_surface_dbuoyb
 33/147 Test #2120: test_gdasapp_C48mx500_3DVarAOWCDA ........................................   Passed   24.44 sec
        Start 2121: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_stage_ic_202103241800
 34/147 Test #2130: test_gdasapp_C48mx500_hybAOWCDA ..........................................   Passed   26.46 sec
        Start 2122: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_waveinit_202103241800
 35/147 Test #2161: test_gdasapp_bufr2ioda_insitu_profile_tesac ..............................   Passed   11.98 sec
        Start 2131: test_gdasapp_C48mx500_hybAOWCDA_gdas_stage_ic_202103241800
 36/147 Test #2166: test_gdasapp_bufr2ioda_insitu_surface_dbuoyb .............................   Passed    3.12 sec
        Start 2133: test_gdasapp_C48mx500_hybAOWCDA_enkfgdas_stage_ic_202103241800
 37/147 Test #2041: test_gdasapp_C96C48_hybatmDA .............................................   Passed   27.73 sec
        Start 2042: test_gdasapp_C96C48_hybatmDA_gdas_stage_ic_202112201800
 38/147 Test #2060: test_gdasapp_C96C48_ufs_hybatmDA .........................................   Passed   28.47 sec
        Start 2061: test_gdasapp_C96C48_ufs_hybatmDA_gdas_stage_ic_202402231800
 39/147 Test #2099: test_gdasapp_C96C48_hybatmsnowDA .........................................   Passed   28.87 sec
        Start 2045: test_gdasapp_C96C48_hybatmDA_enkfgdas_stage_ic_202112201800
 40/147 Test #2146: test_gdasapp_atm_jjob_var_init ...........................................   Passed   78.52 sec
        Start 2064: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_stage_ic_202402231800
 41/147 Test #2150: test_gdasapp_atm_jjob_ens_init ...........................................   Passed   78.63 sec
        Start 2100: test_gdasapp_C96C48_hybatmsnowDA_gdas_stage_ic_202112201200
 42/147 Test #2089: test_gdasapp_C96_gcafs_cycled_gcdas_fetch_202112201800 ...................***Failed   63.54 sec
        Start 2103: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_stage_ic_202112201200
 43/147 Test #2085: test_gdasapp_C96_gcafs_cycled_gcdas_stage_ic_202112201200 ................   Passed   64.48 sec
        Start 2086: test_gdasapp_C96_gcafs_cycled_gcdas_fcst_202112201200
 44/147 Test #2097: test_gdasapp_C96_gcafs_cycled_gcdas_prep_emissions_202112201800 ..........   Passed   63.69 sec
        Start 2151: test_gdasapp_atm_jjob_ens_letkf
 45/147 Test #2133: test_gdasapp_C48mx500_hybAOWCDA_enkfgdas_stage_ic_202103241800 ...........   Passed   59.79 sec
        Start 2134: test_gdasapp_C48mx500_hybAOWCDA_enkfgdas_fcst_202103241800
 46/147 Test #2131: test_gdasapp_C48mx500_hybAOWCDA_gdas_stage_ic_202103241800 ...............   Passed   60.53 sec
        Start 2132: test_gdasapp_C48mx500_hybAOWCDA_gdas_fcst_202103241800
 47/147 Test #2045: test_gdasapp_C96C48_hybatmDA_enkfgdas_stage_ic_202112201800 ..............   Passed   64.46 sec
        Start 2046: test_gdasapp_C96C48_hybatmDA_enkfgdas_fcst_202112201800
 48/147 Test #2061: test_gdasapp_C96C48_ufs_hybatmDA_gdas_stage_ic_202402231800 ..............   Passed   65.03 sec
        Start 2062: test_gdasapp_C96C48_ufs_hybatmDA_gdas_fcst_202402231800
 49/147 Test #2100: test_gdasapp_C96C48_hybatmsnowDA_gdas_stage_ic_202112201200 ..............   Passed   36.00 sec
        Start 2101: test_gdasapp_C96C48_hybatmsnowDA_gdas_fcst_202112201200
 50/147 Test #2042: test_gdasapp_C96C48_hybatmDA_gdas_stage_ic_202112201800 ..................   Passed   92.69 sec
        Start 2043: test_gdasapp_C96C48_hybatmDA_gdas_fcst_202112201800
 51/147 Test #2122: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_waveinit_202103241800 .............   Passed   95.87 sec
        Start 2147: test_gdasapp_atm_jjob_var_run
 52/147 Test #2121: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_stage_ic_202103241800 .............   Passed  100.87 sec
        Start 2123: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_fcst_202103241800
 53/147 Test #2064: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_stage_ic_202402231800 ..........   Passed   46.90 sec
        Start 2065: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_fcst_202402231800
 54/147 Test #2103: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_stage_ic_202112201200 ..........   Passed   53.47 sec
        Start 2104: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_fcst_202112201200
 55/147 Test #2046: test_gdasapp_C96C48_hybatmDA_enkfgdas_fcst_202112201800 ..................   Passed  359.99 sec
        Start 2048: test_gdasapp_C96C48_hybatmDA_enkfgdas_epmn_202112201800
 56/147 Test #2065: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_fcst_202402231800 ..............   Passed  344.40 sec
        Start 2066: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_epmn_202402231800
 57/147 Test #2104: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_fcst_202112201200 ..............   Passed  359.70 sec
        Start 2106: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_epmn_202112201200
 58/147 Test #2132: test_gdasapp_C48mx500_hybAOWCDA_gdas_fcst_202103241800 ...................   Passed  444.96 sec
        Start 2135: test_gdasapp_C48mx500_hybAOWCDA_gdas_prepoceanobs_202103250000
 59/147 Test #2134: test_gdasapp_C48mx500_hybAOWCDA_enkfgdas_fcst_202103241800 ...............   Passed  472.14 sec
        Start 2136: test_gdasapp_C48mx500_hybAOWCDA_gdas_marinebmat_202103250000
 60/147 Test #2066: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_epmn_202402231800 ..............   Passed   93.14 sec
        Start 2167: test_gdasapp_bufr2ioda_insitu_surface_dbuoyb_drifter
 61/147 Test #2167: test_gdasapp_bufr2ioda_insitu_surface_dbuoyb_drifter .....................   Passed    3.87 sec
 62/147 Test #2123: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_fcst_202103241800 .................   Passed  444.24 sec
        Start 2124: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_prepoceanobs_202103250000
        Start 2125: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marinebmat_202103250000
 63/147 Test #2048: test_gdasapp_C96C48_hybatmDA_enkfgdas_epmn_202112201800 ..................   Passed  117.84 sec
 64/147 Test #2106: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_epmn_202112201200 ..............   Passed  122.10 sec
 65/147 Test #2062: test_gdasapp_C96C48_ufs_hybatmDA_gdas_fcst_202402231800 ..................   Passed  530.94 sec
        Start 2063: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmos_prod_202402231800
 66/147 Test #2101: test_gdasapp_C96C48_hybatmsnowDA_gdas_fcst_202112201200 ..................   Passed  522.23 sec
        Start 2102: test_gdasapp_C96C48_hybatmsnowDA_gdas_atmos_prod_202112201200
        Start 2105: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_echgres_202112201200
 67/147 Test #2043: test_gdasapp_C96C48_hybatmDA_gdas_fcst_202112201800 ......................   Passed  528.02 sec
        Start 2044: test_gdasapp_C96C48_hybatmDA_gdas_atmos_prod_202112201800
        Start 2047: test_gdasapp_C96C48_hybatmDA_enkfgdas_echgres_202112201800
 68/147 Test #2125: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marinebmat_202103250000 ...........   Passed   91.00 sec
 69/147 Test #2136: test_gdasapp_C48mx500_hybAOWCDA_gdas_marinebmat_202103250000 .............   Passed  102.31 sec
 70/147 Test #2105: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_echgres_202112201200 ...........   Passed   61.74 sec
 71/147 Test #2047: test_gdasapp_C96C48_hybatmDA_enkfgdas_echgres_202112201800 ...............   Passed  132.06 sec
 72/147 Test #2102: test_gdasapp_C96C48_hybatmsnowDA_gdas_atmos_prod_202112201200 ............   Passed  145.05 sec
        Start 2107: test_gdasapp_C96C48_hybatmsnowDA_gdas_prep_202112201800
 73/147 Test #2044: test_gdasapp_C96C48_hybatmDA_gdas_atmos_prod_202112201800 ................   Passed  144.25 sec
        Start 2049: test_gdasapp_C96C48_hybatmDA_gdas_prep_202112210000
 74/147 Test #2086: test_gdasapp_C96_gcafs_cycled_gcdas_fcst_202112201200 ....................   Passed  708.49 sec
        Start 2087: test_gdasapp_C96_gcafs_cycled_gcdas_atmos_prod_202112201200
        Start 2088: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlgenb_202112201200
        Start 2090: test_gdasapp_C96_gcafs_cycled_gcdas_offlineanl_202112201800
 75/147 Test #2151: test_gdasapp_atm_jjob_ens_letkf ..........................................   Passed  714.33 sec
        Start 2152: test_gdasapp_atm_jjob_ens_init_split
 76/147 Test #2147: test_gdasapp_atm_jjob_var_run ............................................   Passed  682.39 sec
        Start 2148: test_gdasapp_atm_jjob_var_inc
 77/147 Test #2063: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmos_prod_202402231800 ............   Passed  182.15 sec
        Start 2067: test_gdasapp_C96C48_ufs_hybatmDA_gdas_prep_202402240000
 78/147 Test #2152: test_gdasapp_atm_jjob_ens_init_split .....................................   Passed   46.08 sec
        Start 2153: test_gdasapp_atm_jjob_ens_obs
 79/147 Test #2148: test_gdasapp_atm_jjob_var_inc ............................................   Passed   42.22 sec
        Start 2149: test_gdasapp_atm_jjob_var_final
 80/147 Test #2135: test_gdasapp_C48mx500_hybAOWCDA_gdas_prepoceanobs_202103250000 ...........   Passed  323.14 sec
        Start 2138: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlinit_202103250000
        Start 2137: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlletkf_202103250000
 81/147 Test #2124: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_prepoceanobs_202103250000 .........   Passed  312.36 sec
        Start 2126: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlinit_202103250000
 82/147 Test #2149: test_gdasapp_atm_jjob_var_final ..........................................   Passed   42.24 sec
 83/147 Test #2090: test_gdasapp_C96_gcafs_cycled_gcdas_offlineanl_202112201800 ..............***Failed  109.85 sec
        Start 2091: test_gdasapp_C96_gcafs_cycled_gcdas_sfcanl_202112201800
 84/147 Test #2088: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlgenb_202112201200 .............   Passed  115.32 sec
 85/147 Test #2138: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlinit_202103250000 ..........   Passed   58.78 sec
        Start 2139: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlvar_202103250000
 86/147 Test #2153: test_gdasapp_atm_jjob_ens_obs ............................................   Passed   74.25 sec
        Start 2154: test_gdasapp_atm_jjob_ens_sol
 87/147 Test #2087: test_gdasapp_C96_gcafs_cycled_gcdas_atmos_prod_202112201200 ..............   Passed  147.29 sec
        Start 2096: test_gdasapp_C96_gcafs_cycled_gcdas_prepobsaero_202112201800
 88/147 Test #2126: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlinit_202103250000 ........   Passed   65.33 sec
        Start 2127: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlvar_202103250000
 89/147 Test #2154: test_gdasapp_atm_jjob_ens_sol ............................................   Passed   42.23 sec
        Start 2155: test_gdasapp_atm_jjob_ens_inc
 90/147 Test #2139: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlvar_202103250000 ...........   Passed   74.60 sec
        Start 2141: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlchkpt_202103250000
 91/147 Test #2091: test_gdasapp_C96_gcafs_cycled_gcdas_sfcanl_202112201800 ..................***Failed   87.12 sec
 92/147 Test #2096: test_gdasapp_C96_gcafs_cycled_gcdas_prepobsaero_202112201800 .............   Passed   60.96 sec
        Start 2093: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlinit_202112201800
 93/147 Test #2155: test_gdasapp_atm_jjob_ens_inc ............................................   Passed   42.22 sec
        Start 2156: test_gdasapp_atm_jjob_ens_final
 94/147 Test #2156: test_gdasapp_atm_jjob_ens_final ..........................................   Passed   42.22 sec
 95/147 Test #2127: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlvar_202103250000 .........   Passed  108.98 sec
        Start 2128: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlchkpt_202103250000
 96/147 Test #2141: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlchkpt_202103250000 .........   Passed   74.80 sec
        Start 2142: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlfinal_202103250000
 97/147 Test #2093: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlinit_202112201800 .............   Passed   88.39 sec
        Start 2094: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlvar_202112201800
 98/147 Test #2137: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlletkf_202103250000 .........   Passed  237.74 sec
        Start 2140: test_gdasapp_C48mx500_hybAOWCDA_gdas_ocnanalecen_202103250000
 99/147 Test #2142: test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlfinal_202103250000 .........   Passed   60.19 sec
100/147 Test #2107: test_gdasapp_C96C48_hybatmsnowDA_gdas_prep_202112201800 ..................   Passed  343.45 sec
        Start 2108: test_gdasapp_C96C48_hybatmsnowDA_gdas_anal_202112201800
        Start 2113: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_eobs_202112201800
        Start 2109: test_gdasapp_C96C48_hybatmsnowDA_gdas_snowanl_202112201800
101/147 Test #2128: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlchkpt_202103250000 .......   Passed   79.75 sec
        Start 2129: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlfinal_202103250000
102/147 Test #2140: test_gdasapp_C48mx500_hybAOWCDA_gdas_ocnanalecen_202103250000 ............   Passed   60.87 sec
103/147 Test #2049: test_gdasapp_C96C48_hybatmDA_gdas_prep_202112210000 ......................   Passed  372.01 sec
        Start 2054: test_gdasapp_C96C48_hybatmDA_enkfgdas_eobs_202112210000
        Start 2050: test_gdasapp_C96C48_hybatmDA_gdas_anal_202112210000
104/147 Test #2067: test_gdasapp_C96C48_ufs_hybatmDA_gdas_prep_202402240000 ..................   Passed  401.80 sec
        Start 2068: test_gdasapp_C96C48_ufs_hybatmDA_gdas_prepatmiodaobs_202402240000
105/147 Test #2129: test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlfinal_202103250000 .......   Passed   79.75 sec
106/147 Test #2109: test_gdasapp_C96C48_hybatmsnowDA_gdas_snowanl_202112201800 ...............   Passed  114.04 sec
        Start 2117: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_esnowanl_202112201800
107/147 Test #2117: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_esnowanl_202112201800 ..........   Passed  104.59 sec
108/147 Test #2094: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlvar_202112201800 ..............   Passed  336.29 sec
        Start 2095: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlfinal_202112201800
109/147 Test #2095: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlfinal_202112201800 ............   Passed   74.63 sec
        Start 2098: test_gdasapp_C96_gcafs_cycled_gcdas_fcst_202112201800
110/147 Test #2098: test_gdasapp_C96_gcafs_cycled_gcdas_fcst_202112201800 ....................***Failed  161.97 sec
        Start 2092: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlgenb_202112201800
111/147 Test #2092: test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlgenb_202112201800 .............***Failed   60.89 sec
112/147 Test #2113: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_eobs_202112201800 ..............   Passed  950.79 sec
        Start 2114: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_ediag_202112201800
113/147 Test #2068: test_gdasapp_C96C48_ufs_hybatmDA_gdas_prepatmiodaobs_202402240000 ........   Passed  951.39 sec
        Start 2076: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlinit_202402240000
        Start 2069: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlinit_202402240000
114/147 Test #2054: test_gdasapp_C96C48_hybatmDA_enkfgdas_eobs_202112210000 ..................   Passed  1039.01 sec
        Start 2055: test_gdasapp_C96C48_hybatmDA_enkfgdas_ediag_202112210000
115/147 Test #2076: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlinit_202402240000 .....   Passed   87.45 sec
        Start 2077: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlobs_202402240000
116/147 Test #2069: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlinit_202402240000 ............   Passed   87.60 sec
        Start 2070: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlvar_202402240000
117/147 Test #2114: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_ediag_202112201800 .............   Passed  218.89 sec
        Start 2115: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_eupd_202112201800
118/147 Test #2055: test_gdasapp_C96C48_hybatmDA_enkfgdas_ediag_202112210000 .................   Passed  236.86 sec
        Start 2056: test_gdasapp_C96C48_hybatmDA_enkfgdas_eupd_202112210000
119/147 Test #2115: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_eupd_202112201800 ..............   Passed  210.88 sec
120/147 Test #2056: test_gdasapp_C96C48_hybatmDA_enkfgdas_eupd_202112210000 ..................   Passed  230.66 sec
121/147 Test #2050: test_gdasapp_C96C48_hybatmDA_gdas_anal_202112210000 ......................   Passed  1691.06 sec
        Start 2051: test_gdasapp_C96C48_hybatmDA_gdas_sfcanl_202112210000
122/147 Test #2108: test_gdasapp_C96C48_hybatmsnowDA_gdas_anal_202112201800 ..................   Passed  1781.37 sec
        Start 2110: test_gdasapp_C96C48_hybatmsnowDA_gdas_sfcanl_202112201800
123/147 Test #2051: test_gdasapp_C96C48_hybatmDA_gdas_sfcanl_202112210000 ....................   Passed  149.23 sec
        Start 2052: test_gdasapp_C96C48_hybatmDA_gdas_analcalc_202112210000
        Start 2053: test_gdasapp_C96C48_hybatmDA_gdas_fcst_202112210000
124/147 Test #2110: test_gdasapp_C96C48_hybatmsnowDA_gdas_sfcanl_202112201800 ................   Passed  104.35 sec
        Start 2111: test_gdasapp_C96C48_hybatmsnowDA_gdas_analcalc_202112201800
        Start 2112: test_gdasapp_C96C48_hybatmsnowDA_gdas_fcst_202112201800
125/147 Test #2052: test_gdasapp_C96C48_hybatmDA_gdas_analcalc_202112210000 ..................   Passed  101.61 sec
        Start 2057: test_gdasapp_C96C48_hybatmDA_enkfgdas_ecmn_202112210000
        Start 2058: test_gdasapp_C96C48_hybatmDA_enkfgdas_esfc_202112210000
126/147 Test #2111: test_gdasapp_C96C48_hybatmsnowDA_gdas_analcalc_202112201800 ..............   Passed  114.93 sec
        Start 2116: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_ecmn_202112201800
        Start 2118: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_esfc_202112201800
127/147 Test #2057: test_gdasapp_C96C48_hybatmDA_enkfgdas_ecmn_202112210000 ..................   Passed   81.13 sec
128/147 Test #2116: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_ecmn_202112201800 ..............   Passed   86.90 sec
129/147 Test #2118: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_esfc_202112201800 ..............   Passed  175.25 sec
        Start 2119: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_fcst_202112201800
130/147 Test #2058: test_gdasapp_C96C48_hybatmDA_enkfgdas_esfc_202112210000 ..................   Passed  230.41 sec
        Start 2059: test_gdasapp_C96C48_hybatmDA_enkfgdas_fcst_202112210000
131/147 Test #2077: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlobs_202402240000 ......   Passed  1174.84 sec
        Start 2078: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlsol_202402240000
132/147 Test #2078: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlsol_202402240000 ......   Passed  131.36 sec
        Start 2079: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlfv3inc_202402240000
133/147 Test #2079: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlfv3inc_202402240000 ...   Passed   63.28 sec
        Start 2080: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlfinal_202402240000
134/147 Test #2112: test_gdasapp_C96C48_hybatmsnowDA_gdas_fcst_202112201800 ..................   Passed  644.69 sec
135/147 Test #2053: test_gdasapp_C96C48_hybatmDA_gdas_fcst_202112210000 ......................   Passed  665.68 sec
136/147 Test #2119: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_fcst_202112201800 ..............   Passed  402.39 sec
137/147 Test #2059: test_gdasapp_C96C48_hybatmDA_enkfgdas_fcst_202112210000 ..................   Passed  411.94 sec
138/147 Test #2080: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlfinal_202402240000 ....   Passed  303.88 sec
        Start 2082: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_esfc_202402240000
139/147 Test #2082: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_esfc_202402240000 ..............   Passed  175.52 sec
140/147 Test #2070: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlvar_202402240000 .............   Passed  1939.60 sec
        Start 2071: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlfv3inc_202402240000
141/147 Test #2071: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlfv3inc_202402240000 ..........   Passed   62.69 sec
        Start 2072: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlfinal_202402240000
142/147 Test #2072: test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlfinal_202402240000 ...........   Passed  175.26 sec
        Start 2073: test_gdasapp_C96C48_ufs_hybatmDA_gdas_sfcanl_202402240000
        Start 2081: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_ecen_fv3jedi_202402240000
        Start 2074: test_gdasapp_C96C48_ufs_hybatmDA_gdas_analcalc_fv3jedi_202402240000
143/147 Test #2073: test_gdasapp_C96C48_ufs_hybatmDA_gdas_sfcanl_202402240000 ................   Passed   74.32 sec
        Start 2075: test_gdasapp_C96C48_ufs_hybatmDA_gdas_fcst_202402240000
144/147 Test #2074: test_gdasapp_C96C48_ufs_hybatmDA_gdas_analcalc_fv3jedi_202402240000 ......   Passed   87.51 sec
145/147 Test #2081: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_ecen_fv3jedi_202402240000 ......   Passed   93.93 sec
        Start 2083: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_fcst_202402240000
146/147 Test #2083: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_fcst_202402240000 ..............   Passed  357.76 sec
147/147 Test #2075: test_gdasapp_C96C48_ufs_hybatmDA_gdas_fcst_202402240000 ..................   Passed  533.70 sec

97% tests passed, 5 tests failed out of 147

Label Time Summary:
gdas-utils    =  12.63 sec*proc (14 tests)
gdasapp       = 28759.57 sec*proc (114 tests)
script        = 28772.20 sec*proc (128 tests)

Total Test time (real) = 5033.88 sec

The following tests FAILED:
        2089 - test_gdasapp_C96_gcafs_cycled_gcdas_fetch_202112201800 (Failed)
        2090 - test_gdasapp_C96_gcafs_cycled_gcdas_offlineanl_202112201800 (Failed)
        2091 - test_gdasapp_C96_gcafs_cycled_gcdas_sfcanl_202112201800 (Failed)
        2092 - test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlgenb_202112201800 (Failed)
        2098 - test_gdasapp_C96_gcafs_cycled_gcdas_fcst_202112201800 (Failed)
Errors while running CTest
Output from these tests are in: /work2/noaa/da/rtreadon/git/global-workflow/ss_191/sorc/gdas.cd/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

The Hercules results are the same apart from the tests taking longer to run due to longer queue wait times

Test project /work/noaa/da/rtreadon/git/global-workflow/ss_191/sorc/gdas.cd/build
        Start 2060: test_gdasapp_C96C48_ufs_hybatmDA
        Start 2041: test_gdasapp_C96C48_hybatmDA
        Start 2084: test_gdasapp_C96_gcafs_cycled

... 

147/147 Test #2083: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_fcst_202402240000 ..............   Passed  666.86 sec

97% tests passed, 5 tests failed out of 147

Label Time Summary:
gdas-utils    =  26.17 sec*proc (14 tests)
gdasapp       = 106822.95 sec*proc (114 tests)
script        = 106849.13 sec*proc (128 tests)

Total Test time (real) = 12669.96 sec

The following tests FAILED:
        2089 - test_gdasapp_C96_gcafs_cycled_gcdas_fetch_202112201800 (Failed)
        2090 - test_gdasapp_C96_gcafs_cycled_gcdas_offlineanl_202112201800 (Failed)
        2091 - test_gdasapp_C96_gcafs_cycled_gcdas_sfcanl_202112201800 (Failed)
        2092 - test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlgenb_202112201800 (Failed)
        2098 - test_gdasapp_C96_gcafs_cycled_gcdas_fcst_202112201800 (Failed)

The test_gdasapp_C96_gcafs_cycled_gcdas failures are expected. C96_gcafs_cycled does not, at present, run on Orion or Hercules.

Notice that on MSU machines ctest -N -R gdasapp only returns 147 tests. 155 tests are returned on Hera. The 8 missing tests on MSU are

  Test #1650: test_gdasapp_utils_incrhandler
  Test #1651: test_gdasapp_utils_incrhandler_8pes
  Test #1652: test_gdasapp_utils_incrhandler_reproducibility
  Test #1653: test_gdasapp_utils_hybridweights
  Test #1654: test_gdasapp_utils_setcorscales
  Test #1655: test_gdasapp_utils_diagb
  Test #1656: test_gdasapp_utils_enshandler
  Test #1657: test_gdasapp_utils_socatofv3

Logic found in lines 58 to 76 in utils/soca/test/CMakeLists.txt checks for Hera and Ursa. MSU machines wind up in the else() block.

The missing tests would be included on MSU machines if IS_WORKSTATION was set to TRUE. The MSU build did not set this flag to true. Thus, the default, OFF, is used. The Orion and Hercules GDASApp build logs contain the message

-- Skipping GDAS app tests as IS_WORKSTATION is OFF

and the above tests are not included in the build.

In contrast, the Hera GDASApp build log contains the message

-- Detected HPC environment: hfe06

RussTreadon-NOAA · 2025-06-27T16:59:41Z

Ursa ctests

Install feature/spack-stack191 at f4cbf57 inside g-w develop at b60d23e. Most of the GDASApp ctests do not yet run on Ursa. Run a subset of tests via

ctest -j 12 -R gdasapp -E "C96|C48"

with the following results

Test project /scratch3/NCEPDEV/da/Russ.Treadon/git/global-workflow/ss_191/sorc/gdas.cd/build
      Start 2151: test_gdasapp_setup_atm_jjob_cycled_exp
      Start 1651: test_gdasapp_utils_incrhandler_8pes
      Start 1650: test_gdasapp_utils_incrhandler
      Start 1659: test_gdasapp_util_prepdata
      Start 1655: test_gdasapp_utils_diagb
      Start 1657: test_gdasapp_utils_socatofv3
      Start 1656: test_gdasapp_utils_enshandler
      Start 1653: test_gdasapp_utils_hybridweights
      Start 1654: test_gdasapp_utils_setcorscales
      Start 2045: test_gdasapp_snow_apply_jediincr
      Start 2044: test_gdasapp_snow_imsproc
      Start 2150: test_gdasapp_convert_gsi_satbias
 1/41 Test #2151: test_gdasapp_setup_atm_jjob_cycled_exp ...........   Passed    0.75 sec
      Start 2156: test_gdasapp_atm_jjob_ens_init
 2/41 Test #1659: test_gdasapp_util_prepdata .......................   Passed    0.89 sec
      Start 2152: test_gdasapp_atm_jjob_var_init
 3/41 Test #2044: test_gdasapp_snow_imsproc ........................   Passed    3.05 sec
      Start 2041: test_gdasapp_check_python_norms
 4/41 Test #2150: test_gdasapp_convert_gsi_satbias .................   Passed    3.45 sec
      Start 1658: test_gdasapp_util_coding_norms
 5/41 Test #2045: test_gdasapp_snow_apply_jediincr .................   Passed    4.31 sec
      Start 2149: test_gdasapp_convert_bufr_adpsfc
 6/41 Test #2041: test_gdasapp_check_python_norms ..................   Passed    1.92 sec
      Start 2046: test_gdasapp_convert_bufr_adpsfc_snow
 7/41 Test #1654: test_gdasapp_utils_setcorscales ..................   Passed    5.04 sec
      Start 2043: test_gdasapp_snow_create_ens
 8/41 Test #1653: test_gdasapp_utils_hybridweights .................   Passed    5.08 sec
      Start 2163: test_gdasapp_aero_gen_3dvar_yaml
 9/41 Test #1650: test_gdasapp_utils_incrhandler ...................   Passed    5.14 sec
      Start 1660: test_gdasapp_util_rads2ioda
10/41 Test #1651: test_gdasapp_utils_incrhandler_8pes ..............   Passed    5.16 sec
      Start 1652: test_gdasapp_utils_incrhandler_reproducibility
11/41 Test #1657: test_gdasapp_utils_socatofv3 .....................   Passed    5.18 sec
      Start 1664: test_gdasapp_util_viirsaod2ioda
12/41 Test #1656: test_gdasapp_utils_enshandler ....................   Passed    5.19 sec
      Start 1666: test_gdasapp_util_icecamsr2ioda
13/41 Test #1660: test_gdasapp_util_rads2ioda ......................   Passed    0.09 sec
      Start 1662: test_gdasapp_util_smap2ioda
14/41 Test #1664: test_gdasapp_util_viirsaod2ioda ..................   Passed    0.08 sec
      Start 1668: test_gdasapp_util_icecjpssrr2ioda
15/41 Test #1666: test_gdasapp_util_icecamsr2ioda ..................   Passed    0.09 sec
      Start 1663: test_gdasapp_util_smos2ioda
16/41 Test #1655: test_gdasapp_utils_diagb .........................   Passed    5.30 sec
      Start 1665: test_gdasapp_util_icecabi2ioda
17/41 Test #1662: test_gdasapp_util_smap2ioda ......................   Passed    0.09 sec
      Start 1669: test_gdasapp_util_insituall2ioda
18/41 Test #1668: test_gdasapp_util_icecjpssrr2ioda ................   Passed    0.08 sec
      Start 1667: test_gdasapp_util_icecmirs2ioda
19/41 Test #1663: test_gdasapp_util_smos2ioda ......................   Passed    0.08 sec
      Start 2042: test_gdasapp_check_yaml_keys
20/41 Test #2163: test_gdasapp_aero_gen_3dvar_yaml .................   Passed    0.29 sec
      Start 1661: test_gdasapp_util_ghrsst2ioda
21/41 Test #1665: test_gdasapp_util_icecabi2ioda ...................   Passed    0.09 sec
22/41 Test #1669: test_gdasapp_util_insituall2ioda .................   Passed    0.09 sec
23/41 Test #1667: test_gdasapp_util_icecmirs2ioda ..................   Passed    0.09 sec
24/41 Test #2042: test_gdasapp_check_yaml_keys .....................   Passed    0.09 sec
25/41 Test #1661: test_gdasapp_util_ghrsst2ioda ....................   Passed    0.09 sec
26/41 Test #1658: test_gdasapp_util_coding_norms ...................   Passed    2.15 sec
27/41 Test #2043: test_gdasapp_snow_create_ens .....................   Passed    0.61 sec
28/41 Test #1652: test_gdasapp_utils_incrhandler_reproducibility ...   Passed    1.32 sec
29/41 Test #2149: test_gdasapp_convert_bufr_adpsfc .................   Passed    2.26 sec
30/41 Test #2046: test_gdasapp_convert_bufr_adpsfc_snow ............   Passed    1.70 sec
31/41 Test #2156: test_gdasapp_atm_jjob_ens_init ...................   Passed   44.67 sec
      Start 2157: test_gdasapp_atm_jjob_ens_letkf
32/41 Test #2152: test_gdasapp_atm_jjob_var_init ...................   Passed   44.60 sec
      Start 2153: test_gdasapp_atm_jjob_var_run
33/41 Test #2157: test_gdasapp_atm_jjob_ens_letkf ..................   Passed  362.14 sec
      Start 2158: test_gdasapp_atm_jjob_ens_init_split
34/41 Test #2158: test_gdasapp_atm_jjob_ens_init_split .............   Passed   44.56 sec
      Start 2159: test_gdasapp_atm_jjob_ens_obs
35/41 Test #2159: test_gdasapp_atm_jjob_ens_obs ....................   Passed   42.12 sec
      Start 2160: test_gdasapp_atm_jjob_ens_sol
36/41 Test #2153: test_gdasapp_atm_jjob_var_run ....................   Passed  458.14 sec
      Start 2154: test_gdasapp_atm_jjob_var_inc
37/41 Test #2160: test_gdasapp_atm_jjob_ens_sol ....................   Passed   42.15 sec
      Start 2161: test_gdasapp_atm_jjob_ens_inc
38/41 Test #2154: test_gdasapp_atm_jjob_var_inc ....................   Passed   42.11 sec
      Start 2155: test_gdasapp_atm_jjob_var_final
39/41 Test #2161: test_gdasapp_atm_jjob_ens_inc ....................   Passed   42.10 sec
      Start 2162: test_gdasapp_atm_jjob_ens_final
40/41 Test #2155: test_gdasapp_atm_jjob_var_final ..................   Passed   42.11 sec
41/41 Test #2162: test_gdasapp_atm_jjob_ens_final ..................   Passed   42.13 sec

100% tests passed, 0 tests failed out of 41

Label Time Summary:
gdas-utils    =   3.92 sec*proc (12 tests)
gdasapp       = 1207.59 sec*proc (12 tests)
script        = 1211.52 sec*proc (24 tests)

Total Test time (real) = 621.08 sec

All tests run on Ursa Passed.

RussTreadon-NOAA · 2025-06-27T17:56:47Z

Gaea C6 ctests

Install feature/spack-stack191 at f4cbf57 inside g-w develop at b60d23e. Most of the GDASApp ctests do not yet run on Gaea C6. Run a subset of tests via

ctest -j 12 -R gdasapp -E "C96|C48|atm_jjob"

with the following results

Test project /gpfs/f6/ira-sti/scratch/Russ.Treadon/git/global-workflow/ss_191/sorc/gdas.cd/build
      Start 1651: test_gdasapp_util_prepdata
      Start 1650: test_gdasapp_util_coding_norms
      Start 2033: test_gdasapp_check_python_norms
      Start 2034: test_gdasapp_check_yaml_keys
      Start 2035: test_gdasapp_snow_create_ens
      Start 2036: test_gdasapp_snow_imsproc
      Start 2037: test_gdasapp_snow_apply_jediincr
      Start 2038: test_gdasapp_convert_bufr_adpsfc_snow
      Start 2141: test_gdasapp_convert_bufr_adpsfc
      Start 2142: test_gdasapp_convert_gsi_satbias
      Start 2155: test_gdasapp_aero_gen_3dvar_yaml
 1/21 Test #2034: test_gdasapp_check_yaml_keys ............   Passed    0.20 sec
 2/21 Test #2155: test_gdasapp_aero_gen_3dvar_yaml ........   Passed    0.42 sec
 3/21 Test #2035: test_gdasapp_snow_create_ens ............   Passed    0.75 sec
 4/21 Test #1651: test_gdasapp_util_prepdata ..............   Passed    1.09 sec
      Start 1652: test_gdasapp_util_rads2ioda
      Start 1653: test_gdasapp_util_ghrsst2ioda
      Start 1654: test_gdasapp_util_smap2ioda
      Start 1655: test_gdasapp_util_smos2ioda
      Start 1656: test_gdasapp_util_viirsaod2ioda
 5/21 Test #1652: test_gdasapp_util_rads2ioda .............   Passed    0.12 sec
      Start 1657: test_gdasapp_util_icecabi2ioda
 6/21 Test #1654: test_gdasapp_util_smap2ioda .............   Passed    0.12 sec
      Start 1658: test_gdasapp_util_icecamsr2ioda
 7/21 Test #1655: test_gdasapp_util_smos2ioda .............   Passed    0.12 sec
      Start 1659: test_gdasapp_util_icecmirs2ioda
 8/21 Test #1653: test_gdasapp_util_ghrsst2ioda ...........***Exception: SegFault  0.16 sec
      Start 1660: test_gdasapp_util_icecjpssrr2ioda
 9/21 Test #1656: test_gdasapp_util_viirsaod2ioda .........***Exception: SegFault  0.17 sec
      Start 1661: test_gdasapp_util_insituall2ioda
10/21 Test #1658: test_gdasapp_util_icecamsr2ioda .........   Passed    0.12 sec
11/21 Test #1657: test_gdasapp_util_icecabi2ioda ..........   Passed    0.12 sec
12/21 Test #1659: test_gdasapp_util_icecmirs2ioda .........   Passed    0.12 sec
13/21 Test #1660: test_gdasapp_util_icecjpssrr2ioda .......   Passed    0.11 sec
14/21 Test #1661: test_gdasapp_util_insituall2ioda ........   Passed    0.11 sec
15/21 Test #2036: test_gdasapp_snow_imsproc ...............   Passed    1.58 sec
16/21 Test #2038: test_gdasapp_convert_bufr_adpsfc_snow ...   Passed    1.85 sec
17/21 Test #2033: test_gdasapp_check_python_norms .........   Passed    2.22 sec
18/21 Test #1650: test_gdasapp_util_coding_norms ..........   Passed    2.28 sec
19/21 Test #2141: test_gdasapp_convert_bufr_adpsfc ........   Passed    2.44 sec
20/21 Test #2142: test_gdasapp_convert_gsi_satbias ........   Passed    3.02 sec
21/21 Test #2037: test_gdasapp_snow_apply_jediincr ........   Passed    4.97 sec

90% tests passed, 2 tests failed out of 21

Label Time Summary:
gdas-utils    =   4.66 sec*proc (12 tests)
script        =   4.66 sec*proc (12 tests)

Total Test time (real) =   5.33 sec

The following tests FAILED:
        1653 - test_gdasapp_util_ghrsst2ioda (SEGFAULT)
        1656 - test_gdasapp_util_viirsaod2ioda (SEGFAULT)

Rerunning the failed tests with -VV did not provide much additional information (at least not to me). For example, test_gdasapp_util_ghrsst2ioda returned

 Gaea(gaea65):/gpfs/f6/ira-sti/scratch/Russ.Treadon/git/global-workflow/ss_191/sorc/gdas.cd/build$  ctest -VV -R test_gdasapp_util_ghrsst2ioda
UpdateCTestConfiguration  from :/gpfs/f6/ira-sti/scratch/Russ.Treadon/git/global-workflow/ss_191/sorc/gdas.cd/build/DartConfiguration.tcl
Parse Config file:/gpfs/f6/ira-sti/scratch/Russ.Treadon/git/global-workflow/ss_191/sorc/gdas.cd/build/DartConfiguration.tcl
 Add coverage exclude regular expressions.
UpdateCTestConfiguration  from :/gpfs/f6/ira-sti/scratch/Russ.Treadon/git/global-workflow/ss_191/sorc/gdas.cd/build/DartConfiguration.tcl
Parse Config file:/gpfs/f6/ira-sti/scratch/Russ.Treadon/git/global-workflow/ss_191/sorc/gdas.cd/build/DartConfiguration.tcl
Test project /gpfs/f6/ira-sti/scratch/Russ.Treadon/git/global-workflow/ss_191/sorc/gdas.cd/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1653
    Start 1653: test_gdasapp_util_ghrsst2ioda

1653: Test command: /gpfs/f6/ira-sti/scratch/Russ.Treadon/git/global-workflow/ss_191/sorc/gdas.cd/build/bin/gdas_ioda_obsprovider2ioda.x "../testinput/gdas_ghrsst2ioda.yaml"
1653: Working Directory: /gpfs/f6/ira-sti/scratch/Russ.Treadon/git/global-workflow/ss_191/sorc/gdas.cd/build/gdas-utils/test/obsproc
1653: Environment variables:
1653:  OMP_NUM_THREADS=1
1653: Test timeout computed to be: 1500
1653: OOPS Starting 2025-06-27 13:46:30 (UTC-0400)
1653: [TestReference] Comparing to reference file: testref/ghrsst2ioda.test
1653: Relative float tolerance for tests : 1e-06
1653: Absolute float tolerance for tests : 0
1653: [TestReference] Saving Test output to: testoutput/ghrsst2ioda.test
1653: Configuration input file is: ../testinput/gdas_ghrsst2ioda.yaml
1653: Full configuration is:YAMLConfiguration[path=../testinput/gdas_ghrsst2ioda.yaml, root={provider => GHRSST , window begin => 2021-03-24T15:00:00Z , window end => 2021-03-24T21:00:00Z , binning => {stride => 2 , min number of obs => 1} , bounds => {min => -3 , max => 50} , output file => ghrsst_sst_ma_20210324.ioda.nc , input files => (ghrsst_sst_ma_202103241540.nc4,ghrsst_sst_ma_202103241550.nc4) , test => {reference filename => testref/ghrsst2ioda.test , test output filename => testoutput/ghrsst2ioda.test , float relative tolerance => 1e-06}}]
1653: OOPS_STATS ObjectCountHelper started.
1653: OOPS_STATS Run start                                - Runtime:      0.01 sec,  Memory: total:    59.87 Mb, per task: min =    59.87 Mb, max =    59.87 Mb
1653: Run: Starting gdasapp::ObsProvider2IodaApp
1653: --- Window begin: 2021-03-24T15:00:00Z
1653: --- Window end: 2021-03-24T21:00:00Z
1653: --- Input files: [ghrsst_sst_ma_202103241540.nc4,ghrsst_sst_ma_202103241550.nc4]
1653: --- Output files: ghrsst_sst_ma_20210324.ioda.nc
1653: Processing files provided by GHRSST
1653: Reading... ghrsst_sst_ma_202103241540.nc4
1/1 Test #1653: test_gdasapp_util_ghrsst2ioda ....***Exception: SegFault  0.15 sec

0% tests passed, 1 tests failed out of 1

Label Time Summary:
gdas-utils    =   0.15 sec*proc (1 test)
script        =   0.15 sec*proc (1 test)

Total Test time (real) =   0.47 sec

The following tests FAILED:
        1653 - test_gdasapp_util_ghrsst2ioda (SEGFAULT)

RussTreadon-NOAA · 2025-06-27T18:01:57Z

The changes in the PR have been tested on Hera, Orion, and Hercules, GDASApp ctest behavior is as expected. A subset of test were run on Ursa and Gaea C6. The Ursa results are reasonable. There are two unexpected failures on Gaea C6. These failures may pre-date the update to spack-stack/1.9.2`.

The branch for the PR remains feature/spack-stack191 despite the changes being for spack-stack/1.9.2. Version 1.9.1 was the target version when the parent issue, #1283, was opened. As spack-stack/1.9.1 was rolled out to various machines, issues were found resulting in 1.9.2. The branch for this PR will not be renamed to feature/spack-stack/1.9.2.

RussTreadon-NOAA · 2025-06-27T18:02:15Z

The changes in the PR are ready for review.

RussTreadon-NOAA · 2025-06-27T18:07:36Z

NOTE GDASApp PR #1771 will enable more GDASApp ctests to run on Ursa and Gaea C6. Additional Ursa tests will be available when g-w #3798 is completed.

RussTreadon-NOAA · 2025-06-27T18:35:37Z

No tests were run on Dogwood because this PR does not modify the WCOSS2 build or how GDASApp runs on WCOSS2.

RussTreadon-NOAA · 2025-06-27T20:20:16Z

Gaea C6 ctests - debug build

Rebuild GDASApp with -DCMAKE_BUILD_TYPE=Debug. Rerun subset of GDASApp ctests

Gaea(gaea65):/gpfs/f6/ira-sti/scratch/Russ.Treadon/git/global-workflow/ss_191/sorc/gdas.cd/build$ ctest -j 12 -R gdasapp -E "C96|C48|atm_jjob"
Test project /gpfs/f6/ira-sti/scratch/Russ.Treadon/git/global-workflow/ss_191/sorc/gdas.cd/build
      Start 2037: test_gdasapp_snow_apply_jediincr
      Start 1651: test_gdasapp_util_prepdata
      Start 2141: test_gdasapp_convert_bufr_adpsfc
      Start 2038: test_gdasapp_convert_bufr_adpsfc_snow
      Start 2142: test_gdasapp_convert_gsi_satbias
      Start 2036: test_gdasapp_snow_imsproc
      Start 2033: test_gdasapp_check_python_norms
      Start 1650: test_gdasapp_util_coding_norms
      Start 2035: test_gdasapp_snow_create_ens
      Start 2155: test_gdasapp_aero_gen_3dvar_yaml
      Start 2034: test_gdasapp_check_yaml_keys
 1/21 Test #2034: test_gdasapp_check_yaml_keys ............   Passed    0.20 sec
 2/21 Test #2155: test_gdasapp_aero_gen_3dvar_yaml ........   Passed    0.41 sec
 3/21 Test #2035: test_gdasapp_snow_create_ens ............   Passed    0.77 sec
 4/21 Test #1651: test_gdasapp_util_prepdata ..............   Passed    1.14 sec
      Start 1653: test_gdasapp_util_ghrsst2ioda
      Start 1657: test_gdasapp_util_icecabi2ioda
      Start 1659: test_gdasapp_util_icecmirs2ioda
      Start 1652: test_gdasapp_util_rads2ioda
      Start 1660: test_gdasapp_util_icecjpssrr2ioda
 5/21 Test #1653: test_gdasapp_util_ghrsst2ioda ...........   Passed    0.19 sec
      Start 1656: test_gdasapp_util_viirsaod2ioda
 6/21 Test #1660: test_gdasapp_util_icecjpssrr2ioda .......   Passed    0.39 sec
      Start 1661: test_gdasapp_util_insituall2ioda
 7/21 Test #1656: test_gdasapp_util_viirsaod2ioda .........   Passed    0.22 sec
      Start 1655: test_gdasapp_util_smos2ioda
 8/21 Test #1652: test_gdasapp_util_rads2ioda .............   Passed    0.42 sec
      Start 1654: test_gdasapp_util_smap2ioda
 9/21 Test #1657: test_gdasapp_util_icecabi2ioda ..........   Passed    0.43 sec
      Start 1658: test_gdasapp_util_icecamsr2ioda
10/21 Test #1659: test_gdasapp_util_icecmirs2ioda .........   Passed    0.43 sec
11/21 Test #1661: test_gdasapp_util_insituall2ioda ........   Passed    0.17 sec
12/21 Test #1655: test_gdasapp_util_smos2ioda .............   Passed    0.17 sec
13/21 Test #1654: test_gdasapp_util_smap2ioda .............   Passed    0.16 sec
14/21 Test #1658: test_gdasapp_util_icecamsr2ioda .........   Passed    0.17 sec
15/21 Test #2033: test_gdasapp_check_python_norms .........   Passed    2.23 sec
16/21 Test #1650: test_gdasapp_util_coding_norms ..........   Passed    2.26 sec
17/21 Test #2036: test_gdasapp_snow_imsproc ...............   Passed    2.60 sec
18/21 Test #2142: test_gdasapp_convert_gsi_satbias ........   Passed    3.70 sec
19/21 Test #2038: test_gdasapp_convert_bufr_adpsfc_snow ...   Passed    5.34 sec
20/21 Test #2037: test_gdasapp_snow_apply_jediincr ........***Failed    6.37 sec
21/21 Test #2141: test_gdasapp_convert_bufr_adpsfc ........   Passed    8.85 sec

95% tests passed, 1 tests failed out of 21

Label Time Summary:
gdas-utils    =   6.14 sec*proc (12 tests)
script        =   6.14 sec*proc (12 tests)

Total Test time (real) =   9.18 sec

The following tests FAILED:
        2037 - test_gdasapp_snow_apply_jediincr (Failed)

The two tests which previously failed,

        1653 - test_gdasapp_util_ghrsst2ioda (SEGFAULT)
        1656 - test_gdasapp_util_viirsaod2ioda (SEGFAULT)

now pass. Recompiling with Debug changes compiler options with respect to those use for the default RelWithDebInfobuild.

Debug

/opt/cray/pe/craype/2.7.30/bin/CC -DH5_BUILT_AS_DYNAMIC_LIB -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE -D_POSIX_C_SOURCE=200809L -Dioda_SHARED=1 -Dioda_fortran_SHARED=1 -O0 -g -traceback -std=c++17 -fPIE -qopenmp -MD -MT

RelWithDebInfo

/opt/cray/pe/craype/2.7.30/bin/CC -DH5_BUILT_AS_DYNAMIC_LIB -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE -D_POSIX_C_SOURCE=200809L -Dioda_SHARED=1 -Dioda_fortran_SHARED=1 -O2 -g -DNDEBUG -std=c++17 -fPIE -qopenmp -MD -MT

The only evident differnce is level 0 -vs- level 2 optimization. The -traceback option could also alter how instructions are loaded in memory. It's not clear why the Debug executable passes and the RelWithDebInfo executable fails.

test_gdasapp_snow_apply_jediincr which previously passed, now fails. A rerun of this job with -VV returns

2037:  Positive increment skipped (pathway 8) on snow depth exceeding peak threshold
2037:            0
2037: forrtl: error (65): floating invalid
2037: Image              PC                Routine            Line        Source
2037: libpthread-2.31.s  000014E256CA6910  Unknown               Unknown  Unknown
2037: apply_incr.exe     000000000041A90D  apply_incr_noahmp         568  apply_incr_noahmp_snow.f90
2037: apply_incr.exe     000000000041062A  MAIN__                    148  apply_incr_noahmp_snow.f90
2037: apply_incr.exe     000000000040C4FD  Unknown               Unknown  Unknown
2037: libc-2.31.so       000014E2564F224D  __libc_start_main     Unknown  Unknown
2037: apply_incr.exe     000000000040C42A  Unknown               Unknown  Unknown
2037: srun: error: c6n0262: task 2: Aborted (core dumped)
2037: srun: Terminating StepId=209356873.0
2037: slurmstepd: error: *** STEP 209356873.0 ON c6n0262 CANCELLED AT 2025-06-27T14:48:14 ***

Line 568 is the increment(nn) truncation line below

    ! Truncate increments if requested
    if (truncate) then
       do nn = 1, len_land_vec
            increment(nn) = dble(nint(increment(nn) * 1.0d7)) / 1.0d7
        end do
    end if

Via prints find that there is an increment value of -221.032287128931. Multiplying this by 1.0d7 yields -2,210,322,871.28931. This exceeds the maximum range of -2,147,483,648 to +2,147,483,647 for integer(4) variables. Hence the segmentation fault.

RussTreadon-NOAA · 2025-06-27T20:21:08Z

Thank you @DavidNew-NOAA for the review and approval. Merging now.

RussTreadon-NOAA and others added 10 commits March 24, 2025 17:27

initial commit of GDAS hera modulefile to spack-stack/1.9.1 (#1283)

6754f31

Merge branch 'develop' into feature/spack-stack191

382bbe8

Merge branch 'develop' into feature/spack-stack191

43cc91a

Merge branch 'develop' into feature/spack-stack191

95cc984

initial spack-stack/1.9.1 update of GDAS hercules and orion intel mod…

be21ad8

…ulefiles (#1283)

updates to load py-scipy on MSU (#1283)

469cba8

Merge branch 'develop' into feature/spack-stack191

67d0e11

add tar to MSU modulefiles (#1283)

1e44e8b

Merge remote-tracking branch 'upstream/develop' into feature/spack-st…

80cc935

…ack191

Merge branch 'develop' into feature/spack-stack191

574f3c6

RussTreadon-NOAA self-assigned this May 13, 2025

RussTreadon-NOAA added 18 commits May 13, 2025 08:36

Merge branch 'develop' into feature/spack-stack191

8ae1321

Merge branch 'develop' into feature/spack-stack191

608522b

Merge branch 'develop' into feature/spack-stack191

aa4fa72

Merge branch 'develop' into feature/spack-stack191

7548581

Merge branch 'develop' into feature/spack-stack191

8d3152e

Merge branch 'develop' into feature/spack-stack191

88bb0d3

Merge branch 'develop' into feature/spack-stack191

a291551

Merge branch 'develop' into feature/spack-stack191

d5779a6

Merge branch 'develop' into feature/spack-stack191

2e721aa

Merge branch 'develop' into feature/spack-stack191

9d13fb2

Merge branch 'develop' into feature/spack-stack191

09a5e2f

Merge branch 'develop' into feature/spack-stack191

704c579

Merge branch 'develop' into feature/spack-stack191

f10482e

Merge branch 'develop' into feature/spack-stack191

261b066

Merge branch 'develop' into feature/spack-stack191

5144cc4

Merge branch 'develop' into feature/spack-stack191

517b89e

Merge branch 'develop' into feature/spack-stack191

8686487

Merge branch 'develop' into feature/spack-stack191

8cdf4cd

emcbot added the hera-GW-RT-Running Automated testing with global-workflow running on Hera label Jun 24, 2025

emcbot added hera-GW-RT-Passed Automated testing with global-workflow successful on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Jun 24, 2025

RussTreadon-NOAA and others added 2 commits June 24, 2025 18:48

update hercules and ursa modulefiles (#1283)

b350bf2

Merge branch 'develop' into feature/spack-stack191

4cb3669

RussTreadon-NOAA and others added 2 commits June 27, 2025 06:34

Merge branch 'develop' into feature/spack-stack191

a334339

update hera.intel.lua to spack-stack-1.9.2 (#1283)

f4cbf57

RussTreadon-NOAA requested review from CoryMartin-NOAA, DavidHuber-NOAA and DavidNew-NOAA June 27, 2025 18:02

RussTreadon-NOAA mentioned this pull request Jun 27, 2025

Phase 3 test FieldMetadata error #1764

Closed

DavidNew-NOAA approved these changes Jun 27, 2025

View reviewed changes

RussTreadon-NOAA merged commit a8985de into develop Jun 27, 2025
5 checks passed

RussTreadon-NOAA deleted the feature/spack-stack191 branch June 27, 2025 20:21

RussTreadon-NOAA mentioned this pull request Jun 28, 2025

Update JEDI hashes and land-imsproc repo and hash #1775

Merged

7 tasks

TaylorRoper-NOAA mentioned this pull request Jul 2, 2025

Update cloud build environment #1777

Closed

DavidNew-NOAA pushed a commit that referenced this pull request Jan 16, 2026

Update GDASApp to spack-stack/1.9.2 (#1684)

972e999

Conversation

RussTreadon-NOAA commented May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Companion PRs

Issues

Automated CI tests to run in Global Workflow

Uh oh!

RussTreadon-NOAA commented May 13, 2025

Uh oh!

emcbot commented Jun 24, 2025

Uh oh!

DavidHuber-NOAA commented Jun 25, 2025

Uh oh!

RussTreadon-NOAA commented Jun 25, 2025

Uh oh!

RussTreadon-NOAA commented Jun 27, 2025

Uh oh!

RussTreadon-NOAA commented Jun 27, 2025

Uh oh!

RussTreadon-NOAA commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RussTreadon-NOAA commented Jun 27, 2025

Uh oh!

RussTreadon-NOAA commented Jun 27, 2025

Uh oh!

RussTreadon-NOAA commented Jun 27, 2025

Uh oh!

RussTreadon-NOAA commented Jun 27, 2025

Uh oh!

RussTreadon-NOAA commented Jun 27, 2025

Uh oh!

RussTreadon-NOAA commented Jun 27, 2025

Uh oh!

RussTreadon-NOAA commented Jun 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

RussTreadon-NOAA commented May 13, 2025 •

edited

Loading

RussTreadon-NOAA commented Jun 27, 2025 •

edited

Loading