Skip to content

Use submodules rather than ecbuild_bundle git#811

Merged
CoryMartin-NOAA merged 51 commits into
developfrom
feature/submodule
Jan 16, 2024
Merged

Use submodules rather than ecbuild_bundle git#811
CoryMartin-NOAA merged 51 commits into
developfrom
feature/submodule

Conversation

@CoryMartin-NOAA
Copy link
Copy Markdown
Contributor

This PR creates git submodules for each external JEDI library rather than using ecbuild_bundle to get the library source code through git.

There is also a new set of scripts to automate the update process of the submodules (both an update and a git add, in separate scripts).

Peppered through here are also some minor changes for CI testing that are needed due to this change + the global-workflow using submodules.

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor

FYI: users need to load git-lfs (which defaults to git-lfs/3.1.2) on Hercules in order to exercise git-lfs.

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor

Install GDASApp develop and feature/submodule on Hercules inside g-w. Run ctests for both installations. All tests pass in both installations except test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY.

The develop run of the VRFY job failed with

INFO EvaSetup: Running figure driver
Traceback (most recent call last):
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/eva/bin/eva", line 8, in <module>
    sys.exit(main())
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/eva/lib/python3.9/site-packages/eva/eva_driver.py", line 120, in main
    eva(config_file)
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/eva/lib/python3.9/site-packages/eva/eva_driver.py", line 85, in eva
    figure_driver(eva_dict, data_collections, timing, logger)
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/eva/lib/python3.9/site-packages/eva/plotting/batch/base/plot_tools/figure_driver.py", line \
43, in figure_driver
    graphics = graphics_section.get('figure_list')
AttributeError: 'list' object has no attribute 'get'
running eva on evayamls/eva_icec_amsr2_south_2018041512.yaml
Traceback (most recent call last):
  File "/work2/noaa/da/rtreadon/git/global-workflow/test/sorc/gdas.cd/../../sorc/gdas.cd/scripts/exgdas_global_marine_analysis_vrfy.py", line 230, in <mo\
dule>
    subprocess.run(['eva', infile], check=True)
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/eva/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['eva', 'evayamls/eva_icec_amsr2_south_2018041512.yaml']' returned non-zero exit status 1.
+ JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY[1]: postamble JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY 1705068162 1

The feature/submodule run of the VRFY job didn't get this far. It died with

+ JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY[36]: /work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/scripts/e\
xgdas_global_marine_analysis_vrfy.py
Traceback (most recent call last):
  File "/work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/scripts/exgdas_global_marine_analysis_vrf\
y.py", line 22, in <module>
    import gen_eva_obs_yaml
ModuleNotFoundError: No module named 'gen_eva_obs_yaml'
+ JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY[1]: postamble JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY 1705086750 1

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor

FYI

The update to global-workflow develop at c041968 is likely to cause GDASApp jobs to fail. c041968 . resdet and resens have been replaced with resdetatmos and resensatmos. This was easy to update in feature/submodule. Changes to COMROT and ROTDIR may take more time to sort out.

Tagging @CoryMartin-NOAA , @andytangborn , @guillaumevernieres , @AndrewEichmann-NOAA , and @DavidNew-NOAA

@CoryMartin-NOAA
Copy link
Copy Markdown
Contributor Author

@RussTreadon-NOAA the develop VRFY issue I think is being handled by @apchoiCMD . The feature branch issue looks familiar, is this something that got 'unresolved' by a merge commit? Looks like it may be a PYTHONPATH or cmake related problem?

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor

@RussTreadon-NOAA the develop VRFY issue I think is being handled by @apchoiCMD . The feature branch issue looks familiar, is this something that got 'unresolved' by a merge commit? Looks like it may be a PYTHONPATH or cmake related problem?

Agreed regarding a path issue for feature/submodule. I need to first update feature/submodule to handle g-w c041968. The update causes many test_gdasapp to fail.

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor

@RussTreadon-NOAA the develop VRFY issue I think is being handled by @apchoiCMD . The feature branch issue looks familiar, is this something that got 'unresolved' by a merge commit? Looks like it may be a PYTHONPATH or cmake related problem?

Agreed regarding a path issue for feature/submodule. I need to first update feature/submodule to handle g-w c041968. The update causes many test_gdasapp to fail.

The vrfy jobs runs up to the AttributeError: 'list' object has no attribute 'get' failure observed in develop if I explicitly set the path to ush/eva and ush/soca in PYTHONPATH via the TEST section manually added to run_jjobs.sh

##TEST
PYTHONPATH=/work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/ush/eva:/work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/ush/soca:${PYTHONPATH}
##TEST
PYTHONPATH=${HOMEgfs}/ush/python/wxflow/src:${PYTHONPATH}

The PYTHONPATH generated by the ctest is

+ JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY[30]: PYTHONPATH=/work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/ush/soca:/work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/ush/:/work2/noaa/da/rtreadon/git/global-\
workflow/hercules/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/ush/eva:/work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/sorc/gdas/../../ush/python/wxflow/src:/work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/sorc/gdas/ush:/work2/noaa/da/rtreadon/git/global-wor\
kflow/hercules/sorc/gdas.cd/build/gdas/../lib/python3.7:/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/envs/unified-env/install/intel/2021.9.0/py-pybind11-2.8.1-2cs7v5h/lib/python3.10/site-packages:/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/envs/unified-env/install/intel/2021.9.0/bufr-12.0.1-6pgbiv5/lib64/python3.10/site-packages:/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/envs/unified-env/install/intel/2021.9.0/bufr-12.0.1-6pgbiv5/lib/python3.10/site-packages

Note that eva and soca are in the above PTYHTONPATH as

/work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/ush/soca
/work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/ush/eva

It seems the sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/ush piece causes problems. The direct path /sorc/gdas.cd/ush works. Why this sensitivity?

@apchoiCMD
Copy link
Copy Markdown
Collaborator

@RussTreadon-NOAA the develop VRFY issue I think is being handled by @apchoiCMD . The feature branch issue looks familiar, is this something that got 'unresolved' by a merge commit? Looks like it may be a PYTHONPATH or cmake related problem?

@RussTreadon-NOAA I am handling "vrfy" task on hera and it looks like resolved, so I will submit the PR along with checking it on Orion side, too. @CoryMartin-NOAA just wonder have you updated eva on Orion, too?

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor

@apchoiCMD , if you have change(s) you would like me to test on Orion, let me know what to change and I'll give it a try.

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor

Modify HOMEgfs written to config.base via the following change to ush/soca/run_jjobs.py

         # swap a few variables in config.base
-        var2replace = {'HOMEgfs': self.homegfs,
+        self.homegfs_real = os.path.realpath(self.homegfs)
+        var2replace = {'HOMEgfs': self.homegfs_real,
                        'STMP': self.stmp,

With this change the sorc/gdas.cd/sorc/gdas/../../sorc/gdas.cd/ush piece of PYTHONPATH is replaced with sorc/gdas.cd/ush. As a result, test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY runs up to the same error which occurs in develop, namely

INFO DataCollections:   hofxQC::seaIceFraction       (float32) | Min=+0.0000e+00, Max=+7.1078e-01, RMS=+1.6852e-01
INFO DataCollections: --------------------------------------------------------------------------------
INFO EvaSetup: Running figure driver
Traceback (most recent call last):
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/eva/bin/eva", line 8, in <module>
    sys.exit(main())
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/eva/lib/python3.9/site-packages/eva/eva_driver.py", line 120, in main
    eva(config_file)
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/eva/lib/python3.9/site-packages/eva/eva_driver.py", line 85, in eva
    figure_driver(eva_dict, data_collections, timing, logger)
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/eva/lib/python3.9/site-packages/eva/plotting/batch/base/plot_tools/figure_driver.py", lin\
e 43, in figure_driver
    graphics = graphics_section.get('figure_list')
AttributeError: 'list' object has no attribute 'get'
running eva on evayamls/eva_icec_amsr2_south_2018041512.yaml
Traceback (most recent call last):
  File "/work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/scripts/exgdas_global_marine_analysis_vrfy.py", line 230, in <module>
    subprocess.run(['eva', infile], check=True)
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/eva/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['eva', 'evayamls/eva_icec_amsr2_south_2018041512.yaml']' returned non-zero exit status 1.
+ JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY[1]: postamble JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY 1705154009 1

@apchoiCMD is working on a fix for this error.

The above change to ush/soca/run_jjobs.py was committed to feature/submodule at 695447b

All ctest_gdasapp run within global-workflow with this change in place. 51 out of 52 tests pass on Hercules.

(gdasapp) hercules-login-2:/work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/build$ ctest -R test_gdasapp
Test project /work2/noaa/da/rtreadon/git/global-workflow/hercules/sorc/gdas.cd/build
      Start 1325: test_gdasapp_util_coding_norms
 1/52 Test #1325: test_gdasapp_util_coding_norms ........................   Passed    0.81 sec
      Start 1326: test_gdasapp_util_ioda_example
 2/52 Test #1326: test_gdasapp_util_ioda_example ........................   Passed    0.23 sec
      Start 1327: test_gdasapp_util_prepdata
 3/52 Test #1327: test_gdasapp_util_prepdata ............................   Passed    0.79 sec
      Start 1328: test_gdasapp_util_rads2ioda
 4/52 Test #1328: test_gdasapp_util_rads2ioda ...........................   Passed    0.22 sec
      Start 1329: test_gdasapp_util_ghrsst2ioda
 5/52 Test #1329: test_gdasapp_util_ghrsst2ioda .........................   Passed    0.12 sec
      Start 1330: test_gdasapp_util_smap2ioda
 6/52 Test #1330: test_gdasapp_util_smap2ioda ...........................   Passed    0.19 sec
      Start 1331: test_gdasapp_util_smos2ioda
 7/52 Test #1331: test_gdasapp_util_smos2ioda ...........................   Passed    0.13 sec
      Start 1332: test_gdasapp_util_viirsaod2ioda
 8/52 Test #1332: test_gdasapp_util_viirsaod2ioda .......................   Passed    0.22 sec
      Start 1333: test_gdasapp_util_icecamsr2ioda
 9/52 Test #1333: test_gdasapp_util_icecamsr2ioda .......................   Passed    0.21 sec
      Start 1665: test_gdasapp_check_python_norms
10/52 Test #1665: test_gdasapp_check_python_norms .......................   Passed    0.92 sec
      Start 1666: test_gdasapp_check_yaml_keys
11/52 Test #1666: test_gdasapp_check_yaml_keys ..........................   Passed    0.07 sec
      Start 1667: test_gdasapp_jedi_increment_to_fv3
12/52 Test #1667: test_gdasapp_jedi_increment_to_fv3 ....................   Passed    0.45 sec
      Start 1668: test_gdasapp_convert_ewok_yaml
13/52 Test #1668: test_gdasapp_convert_ewok_yaml ........................   Passed    0.25 sec
      Start 1669: test_gdasapp_setup_cycled_exp
14/52 Test #1669: test_gdasapp_setup_cycled_exp .........................   Passed    1.01 sec
      Start 1670: test_gdasapp_convert_bufr_temp_dbuoy
15/52 Test #1670: test_gdasapp_convert_bufr_temp_dbuoy ..................   Passed    0.39 sec
      Start 1671: test_gdasapp_convert_bufr_salt_dbuoy
16/52 Test #1671: test_gdasapp_convert_bufr_salt_dbuoy ..................   Passed    0.40 sec
      Start 1672: test_gdasapp_convert_bufr_temp_mbuoyb
17/52 Test #1672: test_gdasapp_convert_bufr_temp_mbuoyb .................   Passed    0.42 sec
      Start 1673: test_gdasapp_convert_bufr_salt_mbuoyb
18/52 Test #1673: test_gdasapp_convert_bufr_salt_mbuoyb .................   Passed    0.39 sec
      Start 1674: test_gdasapp_convert_bufr_tesacprof
19/52 Test #1674: test_gdasapp_convert_bufr_tesacprof ...................   Passed    0.41 sec
      Start 1675: test_gdasapp_convert_bufr_trkobprof
20/52 Test #1675: test_gdasapp_convert_bufr_trkobprof ...................   Passed    0.38 sec
      Start 1676: test_gdasapp_convert_bufr_sfcships
21/52 Test #1676: test_gdasapp_convert_bufr_sfcships ....................   Passed    0.39 sec
      Start 1677: test_gdasapp_convert_bufr_sfcshipsu
22/52 Test #1677: test_gdasapp_convert_bufr_sfcshipsu ...................   Passed    0.39 sec
      Start 1678: test_gdasapp_soca_nsst_increment_to_mom6
23/52 Test #1678: test_gdasapp_soca_nsst_increment_to_mom6 ..............   Passed   45.03 sec
      Start 1679: test_gdasapp_soca_prep
24/52 Test #1679: test_gdasapp_soca_prep ................................   Passed    7.55 sec
      Start 1680: test_gdasapp_soca_run_clean
25/52 Test #1680: test_gdasapp_soca_run_clean ...........................   Passed    0.25 sec
      Start 1681: test_gdasapp_soca_setup_obsproc
26/52 Test #1681: test_gdasapp_soca_setup_obsproc .......................   Passed    0.83 sec
      Start 1682: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS
27/52 Test #1682: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS ..............   Passed   43.60 sec
      Start 1683: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP
28/52 Test #1683: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP ....   Passed   42.26 sec
      Start 1684: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT
29/52 Test #1684: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT ....   Passed   42.27 sec
      Start 1685: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN
30/52 Test #1685: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN .....   Passed   42.27 sec
      Start 1686: test_gdasapp_soca_copy_scratch
31/52 Test #1686: test_gdasapp_soca_copy_scratch ........................   Passed    2.86 sec
      Start 1687: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT
32/52 Test #1687: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT ...   Passed   42.26 sec
      Start 1688: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST
33/52 Test #1688: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST ....   Passed   42.48 sec
      Start 1689: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY
34/52 Test #1689: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY ....***Failed  367.41 sec
      Start 1690: test_gdasapp_soca_socahybridweights
35/52 Test #1690: test_gdasapp_soca_socahybridweights ...................   Passed  138.30 sec
      Start 1691: test_gdasapp_soca_incr_handler
36/52 Test #1691: test_gdasapp_soca_incr_handler ........................   Passed  138.24 sec
      Start 1692: test_gdasapp_soca_ens_handler
37/52 Test #1692: test_gdasapp_soca_ens_handler .........................   Passed   42.24 sec
      Start 1693: test_gdasapp_land_create_ens
38/52 Test #1693: test_gdasapp_land_create_ens ..........................   Passed    1.97 sec
      Start 1694: test_gdasapp_land_imsproc
39/52 Test #1694: test_gdasapp_land_imsproc .............................   Passed    7.40 sec
      Start 1695: test_gdasapp_land_apply_jediincr
40/52 Test #1695: test_gdasapp_land_apply_jediincr ......................   Passed    2.92 sec
      Start 1696: test_gdasapp_land_letkfoi_snowda
41/52 Test #1696: test_gdasapp_land_letkfoi_snowda ......................   Passed    8.90 sec
      Start 1697: test_gdasapp_convert_bufr_adpsfc_snow
42/52 Test #1697: test_gdasapp_convert_bufr_adpsfc_snow .................   Passed    3.02 sec
      Start 1698: test_gdasapp_convert_bufr_adpsfc
43/52 Test #1698: test_gdasapp_convert_bufr_adpsfc ......................   Passed    2.76 sec
      Start 1699: test_gdasapp_convert_gsi_satbias
44/52 Test #1699: test_gdasapp_convert_gsi_satbias ......................   Passed    1.93 sec
      Start 1700: test_gdasapp_setup_atm_cycled_exp
45/52 Test #1700: test_gdasapp_setup_atm_cycled_exp .....................   Passed    1.85 sec
      Start 1701: test_gdasapp_atm_jjob_var_init
46/52 Test #1701: test_gdasapp_atm_jjob_var_init ........................   Passed   49.95 sec
      Start 1702: test_gdasapp_atm_jjob_var_run
47/52 Test #1702: test_gdasapp_atm_jjob_var_run .........................   Passed   74.34 sec
      Start 1703: test_gdasapp_atm_jjob_var_final
48/52 Test #1703: test_gdasapp_atm_jjob_var_final .......................   Passed   42.39 sec
      Start 1704: test_gdasapp_atm_jjob_ens_init
49/52 Test #1704: test_gdasapp_atm_jjob_ens_init ........................   Passed   22.95 sec
      Start 1705: test_gdasapp_atm_jjob_ens_run
50/52 Test #1705: test_gdasapp_atm_jjob_ens_run .........................   Passed  266.39 sec
      Start 1706: test_gdasapp_atm_jjob_ens_final
51/52 Test #1706: test_gdasapp_atm_jjob_ens_final .......................   Passed   42.39 sec
      Start 1707: test_gdasapp_aero_gen_3dvar_yaml
52/52 Test #1707: test_gdasapp_aero_gen_3dvar_yaml ......................   Passed    0.63 sec

98% tests passed, 1 tests failed out of 52

Label Time Summary:
gdas-utils    =   2.92 sec*proc (9 tests)
script        =   2.92 sec*proc (9 tests)

Total Test time (real) = 1536.89 sec

The following tests FAILED:
        1689 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY (Failed)
Errors while running CTest

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor

I have no more planned work on this PR apart from changes requested or suggested by reviewers.

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor

Orion gw ctests
Install feature/submodule at 695447b in g-w develop at c590476. All ctests pass except the VRFY job

(gdasapp) Orion-login-3:/work2/noaa/da/rtreadon/git/global-workflow/orion/sorc/gdas.cd/build$ ctest -R test_gdasapp
Test project /work2/noaa/da/rtreadon/git/global-workflow/orion/sorc/gdas.cd/build
      Start 1325: test_gdasapp_util_coding_norms
 1/52 Test #1325: test_gdasapp_util_coding_norms ........................   Passed    1.14 sec
      Start 1326: test_gdasapp_util_ioda_example
 2/52 Test #1326: test_gdasapp_util_ioda_example ........................   Passed    2.33 sec
      Start 1327: test_gdasapp_util_prepdata
 3/52 Test #1327: test_gdasapp_util_prepdata ............................   Passed    0.70 sec
      Start 1328: test_gdasapp_util_rads2ioda
 4/52 Test #1328: test_gdasapp_util_rads2ioda ...........................   Passed    0.17 sec
      Start 1329: test_gdasapp_util_ghrsst2ioda
 5/52 Test #1329: test_gdasapp_util_ghrsst2ioda .........................   Passed    0.16 sec
      Start 1330: test_gdasapp_util_smap2ioda
 6/52 Test #1330: test_gdasapp_util_smap2ioda ...........................   Passed    0.16 sec
      Start 1331: test_gdasapp_util_smos2ioda
 7/52 Test #1331: test_gdasapp_util_smos2ioda ...........................   Passed    0.16 sec
      Start 1332: test_gdasapp_util_viirsaod2ioda
 8/52 Test #1332: test_gdasapp_util_viirsaod2ioda .......................   Passed    0.17 sec
      Start 1333: test_gdasapp_util_icecamsr2ioda
 9/52 Test #1333: test_gdasapp_util_icecamsr2ioda .......................   Passed    0.16 sec
      Start 1665: test_gdasapp_check_python_norms
10/52 Test #1665: test_gdasapp_check_python_norms .......................   Passed    1.10 sec
      Start 1666: test_gdasapp_check_yaml_keys
11/52 Test #1666: test_gdasapp_check_yaml_keys ..........................   Passed    0.09 sec
      Start 1667: test_gdasapp_jedi_increment_to_fv3
12/52 Test #1667: test_gdasapp_jedi_increment_to_fv3 ....................   Passed    0.52 sec
      Start 1668: test_gdasapp_convert_ewok_yaml
13/52 Test #1668: test_gdasapp_convert_ewok_yaml ........................   Passed    0.19 sec
      Start 1669: test_gdasapp_setup_cycled_exp
14/52 Test #1669: test_gdasapp_setup_cycled_exp .........................   Passed    0.43 sec
      Start 1670: test_gdasapp_convert_bufr_temp_dbuoy
15/52 Test #1670: test_gdasapp_convert_bufr_temp_dbuoy ..................   Passed    1.96 sec
      Start 1671: test_gdasapp_convert_bufr_salt_dbuoy
16/52 Test #1671: test_gdasapp_convert_bufr_salt_dbuoy ..................   Passed    0.25 sec
      Start 1672: test_gdasapp_convert_bufr_temp_mbuoyb
17/52 Test #1672: test_gdasapp_convert_bufr_temp_mbuoyb .................   Passed    0.25 sec
      Start 1673: test_gdasapp_convert_bufr_salt_mbuoyb
18/52 Test #1673: test_gdasapp_convert_bufr_salt_mbuoyb .................   Passed    0.25 sec
      Start 1674: test_gdasapp_convert_bufr_tesacprof
19/52 Test #1674: test_gdasapp_convert_bufr_tesacprof ...................   Passed    0.26 sec
      Start 1675: test_gdasapp_convert_bufr_trkobprof
20/52 Test #1675: test_gdasapp_convert_bufr_trkobprof ...................   Passed    0.25 sec
      Start 1676: test_gdasapp_convert_bufr_sfcships
21/52 Test #1676: test_gdasapp_convert_bufr_sfcships ....................   Passed    0.26 sec
      Start 1677: test_gdasapp_convert_bufr_sfcshipsu
22/52 Test #1677: test_gdasapp_convert_bufr_sfcshipsu ...................   Passed    0.26 sec
      Start 1678: test_gdasapp_soca_nsst_increment_to_mom6
23/52 Test #1678: test_gdasapp_soca_nsst_increment_to_mom6 ..............   Passed    1.16 sec
      Start 1679: test_gdasapp_soca_prep
24/52 Test #1679: test_gdasapp_soca_prep ................................   Passed    3.21 sec
      Start 1680: test_gdasapp_soca_run_clean
25/52 Test #1680: test_gdasapp_soca_run_clean ...........................   Passed    0.25 sec
      Start 1681: test_gdasapp_soca_setup_obsproc
26/52 Test #1681: test_gdasapp_soca_setup_obsproc .......................   Passed    0.75 sec
      Start 1682: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS
27/52 Test #1682: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS ..............   Passed  139.26 sec
      Start 1683: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP
28/52 Test #1683: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP ....   Passed  298.73 sec
      Start 1684: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT
29/52 Test #1684: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT ....   Passed  332.22 sec
      Start 1685: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN
30/52 Test #1685: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN .....   Passed  299.96 sec
      Start 1686: test_gdasapp_soca_copy_scratch
31/52 Test #1686: test_gdasapp_soca_copy_scratch ........................   Passed    0.92 sec
      Start 1687: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT
32/52 Test #1687: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT ...   Passed  267.32 sec
      Start 1688: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST
33/52 Test #1688: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST ....   Passed  298.30 sec
      Start 1689: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY
34/52 Test #1689: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY ....***Failed  683.07 sec
      Start 1690: test_gdasapp_soca_socahybridweights
35/52 Test #1690: test_gdasapp_soca_socahybridweights ...................   Passed  298.80 sec
      Start 1691: test_gdasapp_soca_incr_handler
36/52 Test #1691: test_gdasapp_soca_incr_handler ........................   Passed  268.11 sec
      Start 1692: test_gdasapp_soca_ens_handler
37/52 Test #1692: test_gdasapp_soca_ens_handler .........................   Passed  333.18 sec
      Start 1693: test_gdasapp_land_create_ens
38/52 Test #1693: test_gdasapp_land_create_ens ..........................   Passed   13.21 sec
      Start 1694: test_gdasapp_land_imsproc
39/52 Test #1694: test_gdasapp_land_imsproc .............................   Passed   18.26 sec
      Start 1695: test_gdasapp_land_apply_jediincr
40/52 Test #1695: test_gdasapp_land_apply_jediincr ......................   Passed   10.03 sec
      Start 1696: test_gdasapp_land_letkfoi_snowda
41/52 Test #1696: test_gdasapp_land_letkfoi_snowda ......................   Passed   31.83 sec
      Start 1697: test_gdasapp_convert_bufr_adpsfc_snow
42/52 Test #1697: test_gdasapp_convert_bufr_adpsfc_snow .................   Passed    6.47 sec
      Start 1698: test_gdasapp_convert_bufr_adpsfc
43/52 Test #1698: test_gdasapp_convert_bufr_adpsfc ......................   Passed    4.42 sec
      Start 1699: test_gdasapp_convert_gsi_satbias
44/52 Test #1699: test_gdasapp_convert_gsi_satbias ......................   Passed    2.15 sec
      Start 1700: test_gdasapp_setup_atm_cycled_exp
45/52 Test #1700: test_gdasapp_setup_atm_cycled_exp .....................   Passed    1.39 sec
      Start 1701: test_gdasapp_atm_jjob_var_init
46/52 Test #1701: test_gdasapp_atm_jjob_var_init ........................   Passed  237.90 sec
      Start 1702: test_gdasapp_atm_jjob_var_run
47/52 Test #1702: test_gdasapp_atm_jjob_var_run .........................   Passed  395.72 sec
      Start 1703: test_gdasapp_atm_jjob_var_final
48/52 Test #1703: test_gdasapp_atm_jjob_var_final .......................   Passed  170.48 sec
      Start 1704: test_gdasapp_atm_jjob_ens_init
49/52 Test #1704: test_gdasapp_atm_jjob_ens_init ........................   Passed   47.50 sec
      Start 1705: test_gdasapp_atm_jjob_ens_run
50/52 Test #1705: test_gdasapp_atm_jjob_ens_run .........................   Passed  938.73 sec
      Start 1706: test_gdasapp_atm_jjob_ens_final
51/52 Test #1706: test_gdasapp_atm_jjob_ens_final .......................   Passed   42.23 sec
      Start 1707: test_gdasapp_aero_gen_3dvar_yaml
52/52 Test #1707: test_gdasapp_aero_gen_3dvar_yaml ......................   Passed    0.57 sec

98% tests passed, 1 tests failed out of 52

Label Time Summary:
gdas-utils    =   5.15 sec*proc (9 tests)
script        =   5.15 sec*proc (9 tests)

Total Test time (real) = 5158.59 sec

The following tests FAILED:
        1689 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY (Failed)
Errors while running CTest
Output from these tests are in: /work2/noaa/da/rtreadon/git/global-workflow/orion/sorc/gdas.cd/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor

Hera gw ctests
Install feature/submodule at 695447b in g-w develop at c590476. All ctests pass except the VRFY job.

(gdasapp) Hera(hfe12):/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/submodule/sorc/gdas.cd/build$ ctest -R test_gdasapp
Test project /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/submodule/sorc/gdas.cd/build
      Start 1325: test_gdasapp_util_coding_norms
 1/52 Test #1325: test_gdasapp_util_coding_norms ........................   Passed    1.38 sec
      Start 1326: test_gdasapp_util_ioda_example
 2/52 Test #1326: test_gdasapp_util_ioda_example ........................   Passed    3.87 sec
      Start 1327: test_gdasapp_util_prepdata
 3/52 Test #1327: test_gdasapp_util_prepdata ............................   Passed    1.41 sec
      Start 1328: test_gdasapp_util_rads2ioda
 4/52 Test #1328: test_gdasapp_util_rads2ioda ...........................   Passed    1.00 sec
      Start 1329: test_gdasapp_util_ghrsst2ioda
 5/52 Test #1329: test_gdasapp_util_ghrsst2ioda .........................   Passed    0.36 sec
      Start 1330: test_gdasapp_util_smap2ioda
 6/52 Test #1330: test_gdasapp_util_smap2ioda ...........................   Passed    0.27 sec
      Start 1331: test_gdasapp_util_smos2ioda
 7/52 Test #1331: test_gdasapp_util_smos2ioda ...........................   Passed    0.15 sec
      Start 1332: test_gdasapp_util_viirsaod2ioda
 8/52 Test #1332: test_gdasapp_util_viirsaod2ioda .......................   Passed    0.17 sec
      Start 1333: test_gdasapp_util_icecamsr2ioda
 9/52 Test #1333: test_gdasapp_util_icecamsr2ioda .......................   Passed    0.19 sec
      Start 1665: test_gdasapp_check_python_norms
10/52 Test #1665: test_gdasapp_check_python_norms .......................   Passed    1.22 sec
      Start 1666: test_gdasapp_check_yaml_keys
11/52 Test #1666: test_gdasapp_check_yaml_keys ..........................   Passed    0.20 sec
      Start 1667: test_gdasapp_jedi_increment_to_fv3
12/52 Test #1667: test_gdasapp_jedi_increment_to_fv3 ....................   Passed    1.82 sec
      Start 1668: test_gdasapp_convert_ewok_yaml
13/52 Test #1668: test_gdasapp_convert_ewok_yaml ........................   Passed    0.69 sec
      Start 1669: test_gdasapp_setup_cycled_exp
14/52 Test #1669: test_gdasapp_setup_cycled_exp .........................   Passed    1.16 sec
      Start 1670: test_gdasapp_convert_bufr_temp_dbuoy
15/52 Test #1670: test_gdasapp_convert_bufr_temp_dbuoy ..................   Passed    2.43 sec
      Start 1671: test_gdasapp_convert_bufr_salt_dbuoy
16/52 Test #1671: test_gdasapp_convert_bufr_salt_dbuoy ..................   Passed    0.33 sec
      Start 1672: test_gdasapp_convert_bufr_temp_mbuoyb
17/52 Test #1672: test_gdasapp_convert_bufr_temp_mbuoyb .................   Passed    0.19 sec
      Start 1673: test_gdasapp_convert_bufr_salt_mbuoyb
18/52 Test #1673: test_gdasapp_convert_bufr_salt_mbuoyb .................   Passed    0.19 sec
      Start 1674: test_gdasapp_convert_bufr_tesacprof
19/52 Test #1674: test_gdasapp_convert_bufr_tesacprof ...................   Passed    0.20 sec
      Start 1675: test_gdasapp_convert_bufr_trkobprof
20/52 Test #1675: test_gdasapp_convert_bufr_trkobprof ...................   Passed    0.19 sec
      Start 1676: test_gdasapp_convert_bufr_sfcships
21/52 Test #1676: test_gdasapp_convert_bufr_sfcships ....................   Passed    0.24 sec
      Start 1677: test_gdasapp_convert_bufr_sfcshipsu
22/52 Test #1677: test_gdasapp_convert_bufr_sfcshipsu ...................   Passed    0.28 sec
      Start 1678: test_gdasapp_soca_nsst_increment_to_mom6
23/52 Test #1678: test_gdasapp_soca_nsst_increment_to_mom6 ..............   Passed   11.44 sec
      Start 1679: test_gdasapp_soca_prep
24/52 Test #1679: test_gdasapp_soca_prep ................................   Passed    3.86 sec
      Start 1680: test_gdasapp_soca_run_clean
25/52 Test #1680: test_gdasapp_soca_run_clean ...........................   Passed    0.55 sec
      Start 1681: test_gdasapp_soca_setup_obsproc
26/52 Test #1681: test_gdasapp_soca_setup_obsproc .......................   Passed    0.64 sec
      Start 1682: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS
27/52 Test #1682: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS ..............   Passed  427.15 sec
      Start 1683: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP
28/52 Test #1683: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP ....   Passed   42.51 sec
      Start 1684: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT
29/52 Test #1684: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT ....   Passed   42.26 sec
      Start 1685: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN
30/52 Test #1685: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN .....   Passed   42.29 sec
      Start 1686: test_gdasapp_soca_copy_scratch
31/52 Test #1686: test_gdasapp_soca_copy_scratch ........................   Passed    1.36 sec
      Start 1687: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT
32/52 Test #1687: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT ...   Passed   42.81 sec
      Start 1688: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST
33/52 Test #1688: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST ....   Passed   44.22 sec
      Start 1689: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY
34/52 Test #1689: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY ....***Failed  300.87 sec
      Start 1690: test_gdasapp_soca_socahybridweights
35/52 Test #1690: test_gdasapp_soca_socahybridweights ...................   Passed  107.88 sec
      Start 1691: test_gdasapp_soca_incr_handler
36/52 Test #1691: test_gdasapp_soca_incr_handler ........................   Passed  139.10 sec
      Start 1692: test_gdasapp_soca_ens_handler
37/52 Test #1692: test_gdasapp_soca_ens_handler .........................   Passed   74.25 sec
      Start 1693: test_gdasapp_land_create_ens
38/52 Test #1693: test_gdasapp_land_create_ens ..........................   Passed    2.70 sec
      Start 1694: test_gdasapp_land_imsproc
39/52 Test #1694: test_gdasapp_land_imsproc .............................   Passed    6.20 sec
      Start 1695: test_gdasapp_land_apply_jediincr
40/52 Test #1695: test_gdasapp_land_apply_jediincr ......................   Passed    5.06 sec
      Start 1696: test_gdasapp_land_letkfoi_snowda
41/52 Test #1696: test_gdasapp_land_letkfoi_snowda ......................   Passed   13.45 sec
      Start 1697: test_gdasapp_convert_bufr_adpsfc_snow
42/52 Test #1697: test_gdasapp_convert_bufr_adpsfc_snow .................   Passed    4.59 sec
      Start 1698: test_gdasapp_convert_bufr_adpsfc
43/52 Test #1698: test_gdasapp_convert_bufr_adpsfc ......................   Passed    4.34 sec
      Start 1699: test_gdasapp_convert_gsi_satbias
44/52 Test #1699: test_gdasapp_convert_gsi_satbias ......................   Passed    1.45 sec
      Start 1700: test_gdasapp_setup_atm_cycled_exp
45/52 Test #1700: test_gdasapp_setup_atm_cycled_exp .....................   Passed    1.03 sec
      Start 1701: test_gdasapp_atm_jjob_var_init
46/52 Test #1701: test_gdasapp_atm_jjob_var_init ........................   Passed  109.07 sec
      Start 1702: test_gdasapp_atm_jjob_var_run
47/52 Test #1702: test_gdasapp_atm_jjob_var_run .........................   Passed  395.71 sec
      Start 1703: test_gdasapp_atm_jjob_var_final
48/52 Test #1703: test_gdasapp_atm_jjob_var_final .......................   Passed   74.15 sec
      Start 1704: test_gdasapp_atm_jjob_ens_init
49/52 Test #1704: test_gdasapp_atm_jjob_ens_init ........................   Passed   43.34 sec
      Start 1705: test_gdasapp_atm_jjob_ens_run
50/52 Test #1705: test_gdasapp_atm_jjob_ens_run .........................   Passed  266.65 sec
      Start 1706: test_gdasapp_atm_jjob_ens_final
51/52 Test #1706: test_gdasapp_atm_jjob_ens_final .......................   Passed  106.16 sec
      Start 1707: test_gdasapp_aero_gen_3dvar_yaml
52/52 Test #1707: test_gdasapp_aero_gen_3dvar_yaml ......................   Passed    1.02 sec

98% tests passed, 1 tests failed out of 52

Label Time Summary:
gdas-utils    =   8.80 sec*proc (9 tests)
script        =   8.80 sec*proc (9 tests)

Total Test time (real) = 2336.14 sec

The following tests FAILED:
        1689 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY (Failed)
Errors while running CTest
Output from these tests are in: /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/submodule/sorc/gdas.cd/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
(gdasapp) Hera(hfe12):/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/submodule/sorc/gdas.cd/build$

As mentioned before the Hecules, Hera, and Orion g-w snapshots contain the following local modifications

  • jobs/JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT - replace ocnanalrun, a typo, with ocnanalbmat
  • parm/config/gfs/config.resources - set nth_ocnanalrun to one thread
  • sorc/build_all.sh - allow GDASApp build on hercules and hera.intel

Note: build_all.sh executes detect_machine.sh to set MACHINE. Script detect_machine.sh includes the compiler with the machine id for hera

# Append compiler (only on machines that have multiple compilers)
if [ $MACHINE_ID = hera ] || [ $MACHINE_ID = cheyenne ]; then
    MACHINE_ID=${MACHINE_ID}.${COMPILER}
fi

As such, detect_machine.sh returns hera.intel when build_all.sh is executed on Hera. The existing logic in build_all.sh did not allow for hera.intel to be a valid machine.

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor

global-workflow branch RussTreadon-NOAA:feature/gdasapp has been created to bring the changes in this PR into the g-w UFS-DA build. Once this PR is merged into GDASApp develop, the hash for gdas.cd in feature/gdasapp will be updated to the new GDASApp develop hash

See g-w issue #2222 for details.

@RussTreadon-NOAA RussTreadon-NOAA self-requested a review January 16, 2024 11:31
Copy link
Copy Markdown
Contributor

@RussTreadon-NOAA RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

feature/submodule installed on Hera, Hercules, and Orion within g-w develop. g-w ctest_gdasapp run on each machine. All tests pass except test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY . A fix for this ctest is under development.

Approve.

Copy link
Copy Markdown
Contributor

@guillaumevernieres guillaumevernieres left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@CoryMartin-NOAA
Copy link
Copy Markdown
Contributor Author

I will merge this later today!

@CoryMartin-NOAA CoryMartin-NOAA merged commit 639ef16 into develop Jan 16, 2024
@CoryMartin-NOAA CoryMartin-NOAA deleted the feature/submodule branch January 16, 2024 15:42
@AndrewEichmann-NOAA
Copy link
Copy Markdown
Collaborator

also resolves #835

DavidNew-NOAA pushed a commit that referenced this pull request Jan 16, 2026
This PR creates git submodules for each external JEDI library rather
than using `ecbuild_bundle` to get the library source code through git.

There is also a new set of scripts to automate the update process of the
submodules (both an update and a git add, in separate scripts).

Peppered through here are also some minor changes for CI testing that
are needed due to this change + the global-workflow using submodules.

---------

Co-authored-by: Guillaume Vernieres <guillaume.vernieres@noaa.gov>
Co-authored-by: RussTreadon-NOAA <russ.treadon@noaa.gov>
Co-authored-by: RussTreadon-NOAA <26926959+RussTreadon-NOAA@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hera-GW-RT-Passed Automated testing with global-workflow successful on Hera

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants