Add 3denvar option to run_jedi_exe.py

[RussTreadon-NOAA]

The changes in this PR add the option to launch fv3jedi_var.x in 3denvar mode via ush/run_jedi_exe.py. This functionality is useful since operational GSI-based global and regional DA systems run in a hybrid model in which the static background error is augmented with an ensemble component (envar).

This PR adds four new files:

• parm/atm/berror/hybvar_gsibec.yaml - hybrid background error yaml file using gsibec for static-B along with a 10 member ensemble
• test/atm/run_jedi_exe_3denvar.sh - ctest using run_jedi_exe.py to launch fv3jedi_var.x in 3denvar mode
• ush/examples/run_jedi_exe/3denvar_hera.yaml - configuration file used by run_jedi_exe.py to run 3denvar sample case on Hera
• ush/examples/run_jedi_exe/3denvar_orion.yaml - configuration file used by run_jedi_exe.py to run 3denvar sample case on Orion

Eleven existing files are modified:

• CMakeLists.txt - update gsibec to tag 1.0.7. Previously pointed at head of gsibec develop. This is problematic since the head of gsibec develop is not always function and/or requires specific hashes of other JEDI components.
• parm/atm/obs/config/sondes.yaml - replace surface_geometric_height with surface_geopotential_height to avoid fv3jedi_var.x segmentation fault
• test/atm/CMakeLists.txt - add test_gdasapp_run_jedi_exe_3denvar
• test/genYAML_prep.sh - replace CASE_ENKF with CASE_ANL
• test/genYAML_prep_aero.sh - replace CASE_ENKF with CASE_ANL
• test/genYAML_prep_land.sh - replace CASE_ENKF with CASE_ANL
• ush/run_jedi_exe.py - remove atmos from enkfgdas root path. This change is necessitate by changes in the g-w GFS directory structure. The atmos directory is now located in memXXX directories.
• ush/ufsda/genYAML.py - replace CASE_ENKF with CASE_ANL
• ush/ufsda/misc_utils.py - replace CASE_ENKF with CASE_ANL
• ush/ufsda/stage.py - add atmos to the memXXX path for ensemble members, enhance error checking for dohybvar = .true. option, add logic to ensure correct gsibec static-B is linked to the run directory.
• ush/ufsda/ufs_yaml.py - replace CASE_ENKF with CASE_ANL

Fixes #402

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Fri Mar 24 16:19:55 UTC 2023 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Fri Mar 24 17:21:14 UTC 2023
---------------------------------------------------
Tests:                                 *SUCCESS*
Tests: Completed at Fri Mar 24 17:35:23 UTC 2023
Tests: 100% tests passed, 0 tests failed out of 49

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: orion

Start: Fri Mar 24 11:19:09 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Fri Mar 24 12:20:23 CDT 2023
---------------------------------------------------
Tests:                                 *SUCCESS*
Tests: Completed at Fri Mar 24 12:38:27 CDT 2023
Tests: 100% tests passed, 0 tests failed out of 49

[kevindougherty-noaa]

Just a couple of quick remarks. I will take a more in-depth look first thing Monday.

[RussTreadon-NOAA]

Rebuilt GDASApp 3/27 evening. Much to my surprise 3dvar and 3denvar jobs launched by run_jedi_exe.py failed.

The GDASApp build clones the head of gsibec develop.

 # Build GSI-B
   option(BUILD_GSIBEC "Build GSI-B" OFF)
   if(BUILD_GSIBEC)
    ecbuild_bundle( PROJECT gsibec GIT "https://github.com/GEOS-ESM/GSIbec.git" BRANCH develop )
   endif()

I reached out to Ricardo Todling (GMAO). He advised that we use the gsibec hash referenced in saber. This is gsibec tag 1.0.7. Update GDASApp top level CMakeLists.txt to clone

 # Build GSI-B
   option(BUILD_GSIBEC "Build GSI-B" OFF)
   if(BUILD_GSIBEC)
    ecbuild_bundle( PROJECT gsibec GIT "https://github.com/GEOS-ESM/GSIbec.git" TAG 1.0.7 )
   endif()

Rebuilt GDASApp. The 3dvar and 3denvar tests successfully ran to completion with gsibec tag 1.0.7. Commit this change feature/3denvar at 018cf8b.

[CoryMartin-NOAA]

Thanks @RussTreadon-NOAA for all the hard work on this and the back and forth in review, much appreciated!

[CoryMartin-NOAA]

Will wait either for a 2nd approval or confirmation about the RT results being as expected before merging

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Wed Apr  5 12:35:32 UTC 2023 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Wed Apr  5 13:31:31 UTC 2023
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Wed Apr  5 13:40:54 UTC 2023
Tests: 98% tests passed, 1 tests failed out of 49
	1417 - test_gdasapp_atm_jjob_ens_run (Failed)
Tests: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/405/global-workflow/sorc/gdas.cd/build/log.ctest

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: orion

Start: Wed Apr  5 07:32:27 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Wed Apr  5 08:38:05 CDT 2023
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Wed Apr  5 08:49:49 CDT 2023
Tests: 98% tests passed, 1 tests failed out of 49
	1417 - test_gdasapp_atm_jjob_ens_run (Failed)
Tests: see output at /work2/noaa/stmp/cmartin/CI/GDASApp/workflow/PR/405/global-workflow/sorc/gdas.cd/build/log.ctest

[CoryMartin-NOAA]

These two single test failures are expected given earlier discussion with @RussTreadon-NOAA , I will merge this PR in and hope that the g-w PR can be merged relatively soon.

[RussTreadon-NOAA]

The Hera and Orion GW-RT failures of test_gdasapp_atm_jjob_ens_run are expected.

This PR removes RESTART from the run directory paths to the ensemble member backgrounds for ensemble DA. This was done to be consistent with the run directory path to the deterministic backgrounds for variational DA. The variational DA path does not include RESTART. The run directory paths to the ensemble members for ensemble variational DA also do not include RESTART.

g-w scripts/exgdas_global_atmos_ensanal_run.sh needs to be updated to account for this change. The required change will be made in g-w PR #1373. Ideally g-w PR #1373 and GDASApp PR #405 would be merged into their respective develop branches around the same time.

[CoryMartin-NOAA]

Sorry @RussTreadon-NOAA if I jumped the gun on merging this. Let me try to review NOAA-EMC/global-workflow#1373 later today and see if we can get the ball rolling on that.

[RussTreadon-NOAA]

Thank you @CoryMartin-NOAA for merging this PR into GDASApp develop.

It's OK to pause your review of g-w PR #1373. g-w #1370 and #1372 are ahead of #1373. EIB is currently reviewing PR #1370.

Now that GDASApp develop has been updated, I'll formally update g-w PR #1373.