Update nc suffix and bufr file path for atmospheric observation yamls and bufr2ioda converters

[RussTreadon-NOAA]

This PR includes two sets of changes related to atmospheric observations:

1. replace .nc4 with .nc in parm/atm/obs/config
2. add atmos to the path for bufr dump files in ush/ioda/bufr2ioda

Resolves #911 (item 1)
Resolves #919 (item 2)

The above changes have been tested and are working in a Hera 3DEnVar JEDI parallel.

[RussTreadon-NOAA]

Install RussTreadon-NOAA:feature/obs at c2d8235 on Hera inside g-w develop at 6f9afff. Execute test_gdasapp inside g-w with all tests except the following passing

88% tests passed, 6 tests failed out of 52

Label Time Summary:
gdas-utils    =   4.02 sec*proc (9 tests)
script        =   4.02 sec*proc (9 tests)

Total Test time (real) = 3144.96 sec

The following tests FAILED:
        1731 - test_gdasapp_atm_jjob_var_init (Failed)
        1732 - test_gdasapp_atm_jjob_var_run (Failed)
        1733 - test_gdasapp_atm_jjob_var_final (Failed)
        1734 - test_gdasapp_atm_jjob_ens_init (Failed)
        1735 - test_gdasapp_atm_jjob_ens_run (Failed)
        1736 - test_gdasapp_atm_jjob_ens_final (Failed)

A check of the var and ens init jobs show that the driver scripts reference .nc4. This is correct since this is how the input data are stored in /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/data/lowres/. However, the updated yamls expect .nc.

Modify jjob_ens_init.sh and jjob_var_init.sh in test/atm/global-workflow to link source .nc4 files to local .nc files. This alone, however, is not sufficient. g-w script analysis.py uses .nc4 for radiance bias correction files. Change a local working copy of this script to use .nc.

After these changes test_gdasapp_atm_jjob_init passed. test_gdasapp_atm_jjob_run failed due to a mismatch between the layout_x and layout_y specified in config.atmanl and config.resources. Additional changes are needed in the working copy of g-w or GDASApp. Further investigation is necessary to determine the best repo in which to make changes.

Regardless of where layout changes are made, it's clear that this GDASApp PR necessitates a g-w PR to update analysis.py. The same g-w PR should also replace the diagnostic file suffix .nc4 with .nc in atm_analysis.py.

A two step approach is proposed

1. merge this PR into GDASApp develop when ready after peer review and approval. The g-w test_gdasapp_atm_jjob tests will fail.
2. open a g-w PR to update analysis.py, atm_analysis.py, the GDASApp hash, and, possibly, select config files. The g-w test_gdasapp_atm_jjob tests will pass.

[RussTreadon-NOAA]

The initialization scripts used for the JEDI ATM variational and local ensemble ctests were updated in feature/obs at 267687c. The same commit also generalized specification of {layout_x, layout_y} used in the variational and local ensemble jobs.

The above described changes to the g-w python scripts which drive the JEDI ATM var and ens apps were commited to forked g-w branch feature/jediatm_nc at 62abf88. This commit also generalized specification of {layout_x, layout_y} for JEDI ATM var and ens apps.

g-w was built from feature/jediatm_nc and feature/obs on Hera. The entire suite of 52 GDASApp g-w ctests was run with the following results.

(gdasapp) Hera(hfe02):/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/jediatm_nc/sorc/gdas.cd/build$ ctest -R test_gdasapp
Test project /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/jediatm_nc/sorc/gdas.cd/build
      Start 1356: test_gdasapp_util_coding_norms
 1/52 Test #1356: test_gdasapp_util_coding_norms ........................   Passed    1.50 sec
      Start 1357: test_gdasapp_util_ioda_example
 2/52 Test #1357: test_gdasapp_util_ioda_example ........................   Passed    6.41 sec
      Start 1358: test_gdasapp_util_prepdata
 3/52 Test #1358: test_gdasapp_util_prepdata ............................   Passed    0.78 sec
      Start 1359: test_gdasapp_util_rads2ioda
 4/52 Test #1359: test_gdasapp_util_rads2ioda ...........................   Passed    0.51 sec
      Start 1360: test_gdasapp_util_ghrsst2ioda
 5/52 Test #1360: test_gdasapp_util_ghrsst2ioda .........................   Passed    0.14 sec
      Start 1361: test_gdasapp_util_smap2ioda
 6/52 Test #1361: test_gdasapp_util_smap2ioda ...........................   Passed    0.13 sec
      Start 1362: test_gdasapp_util_smos2ioda
 7/52 Test #1362: test_gdasapp_util_smos2ioda ...........................   Passed    0.15 sec
      Start 1363: test_gdasapp_util_viirsaod2ioda
 8/52 Test #1363: test_gdasapp_util_viirsaod2ioda .......................   Passed    0.16 sec
      Start 1364: test_gdasapp_util_icecamsr2ioda
 9/52 Test #1364: test_gdasapp_util_icecamsr2ioda .......................   Passed    0.14 sec
      Start 1695: test_gdasapp_check_python_norms
10/52 Test #1695: test_gdasapp_check_python_norms .......................   Passed    1.58 sec
      Start 1696: test_gdasapp_check_yaml_keys
11/52 Test #1696: test_gdasapp_check_yaml_keys ..........................   Passed    0.15 sec
      Start 1697: test_gdasapp_jedi_increment_to_fv3
12/52 Test #1697: test_gdasapp_jedi_increment_to_fv3 ....................   Passed    1.39 sec
      Start 1698: test_gdasapp_convert_ewok_yaml
13/52 Test #1698: test_gdasapp_convert_ewok_yaml ........................   Passed    0.29 sec
      Start 1699: test_gdasapp_setup_cycled_exp
14/52 Test #1699: test_gdasapp_setup_cycled_exp .........................   Passed    1.21 sec
      Start 1700: test_gdasapp_convert_bufr_temp_dbuoy
15/52 Test #1700: test_gdasapp_convert_bufr_temp_dbuoy ..................   Passed    2.72 sec
      Start 1701: test_gdasapp_convert_bufr_salt_dbuoy
16/52 Test #1701: test_gdasapp_convert_bufr_salt_dbuoy ..................   Passed    0.37 sec
      Start 1702: test_gdasapp_convert_bufr_temp_mbuoyb
17/52 Test #1702: test_gdasapp_convert_bufr_temp_mbuoyb .................   Passed    0.25 sec
      Start 1703: test_gdasapp_convert_bufr_salt_mbuoyb
18/52 Test #1703: test_gdasapp_convert_bufr_salt_mbuoyb .................   Passed    0.25 sec
      Start 1704: test_gdasapp_convert_bufr_tesacprof
19/52 Test #1704: test_gdasapp_convert_bufr_tesacprof ...................   Passed    0.25 sec
      Start 1705: test_gdasapp_convert_bufr_trkobprof
20/52 Test #1705: test_gdasapp_convert_bufr_trkobprof ...................   Passed    0.19 sec
      Start 1706: test_gdasapp_convert_bufr_sfcships
21/52 Test #1706: test_gdasapp_convert_bufr_sfcships ....................   Passed    0.20 sec
      Start 1707: test_gdasapp_convert_bufr_sfcshipsu
22/52 Test #1707: test_gdasapp_convert_bufr_sfcshipsu ...................   Passed    0.20 sec
      Start 1708: test_gdasapp_soca_nsst_increment_to_mom6
23/52 Test #1708: test_gdasapp_soca_nsst_increment_to_mom6 ..............   Passed   10.43 sec
      Start 1709: test_gdasapp_soca_prep
24/52 Test #1709: test_gdasapp_soca_prep ................................   Passed    3.05 sec
      Start 1710: test_gdasapp_soca_run_clean
25/52 Test #1710: test_gdasapp_soca_run_clean ...........................   Passed    0.19 sec
      Start 1711: test_gdasapp_soca_setup_obsprep
26/52 Test #1711: test_gdasapp_soca_setup_obsprep .......................   Passed    4.27 sec
      Start 1712: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS
27/52 Test #1712: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS ..............   Passed   46.36 sec
      Start 1713: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP
28/52 Test #1713: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP ....   Passed   43.15 sec
      Start 1714: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT
29/52 Test #1714: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT ....   Passed   75.48 sec
      Start 1715: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN
30/52 Test #1715: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN .....   Passed   43.58 sec
      Start 1716: test_gdasapp_soca_copy_scratch
31/52 Test #1716: test_gdasapp_soca_copy_scratch ........................   Passed    2.29 sec
      Start 1717: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT
32/52 Test #1717: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT ...   Passed   43.03 sec
      Start 1718: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST
33/52 Test #1718: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST ....   Passed   10.24 sec
      Start 1719: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY
34/52 Test #1719: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY ....   Passed  202.55 sec
      Start 1720: test_gdasapp_soca_socahybridweights
35/52 Test #1720: test_gdasapp_soca_socahybridweights ...................   Passed   10.26 sec
      Start 1721: test_gdasapp_soca_incr_handler
36/52 Test #1721: test_gdasapp_soca_incr_handler ........................   Passed   10.67 sec
      Start 1722: test_gdasapp_soca_ens_handler
37/52 Test #1722: test_gdasapp_soca_ens_handler .........................   Passed   10.22 sec
      Start 1723: test_gdasapp_snow_create_ens
38/52 Test #1723: test_gdasapp_snow_create_ens ..........................   Passed    1.15 sec
      Start 1724: test_gdasapp_snow_imsproc
39/52 Test #1724: test_gdasapp_snow_imsproc .............................   Passed    5.18 sec
      Start 1725: test_gdasapp_snow_apply_jediincr
40/52 Test #1725: test_gdasapp_snow_apply_jediincr ......................   Passed    2.94 sec
      Start 1726: test_gdasapp_snow_letkfoi_snowda
41/52 Test #1726: test_gdasapp_snow_letkfoi_snowda ......................   Passed    4.47 sec
      Start 1727: test_gdasapp_convert_bufr_adpsfc_snow
42/52 Test #1727: test_gdasapp_convert_bufr_adpsfc_snow .................   Passed    3.44 sec
      Start 1728: test_gdasapp_convert_bufr_adpsfc
43/52 Test #1728: test_gdasapp_convert_bufr_adpsfc ......................   Passed    4.06 sec
      Start 1729: test_gdasapp_convert_gsi_satbias
44/52 Test #1729: test_gdasapp_convert_gsi_satbias ......................   Passed    1.40 sec
      Start 1730: test_gdasapp_setup_atm_cycled_exp
45/52 Test #1730: test_gdasapp_setup_atm_cycled_exp .....................   Passed    0.86 sec
      Start 1731: test_gdasapp_atm_jjob_var_init
46/52 Test #1731: test_gdasapp_atm_jjob_var_init ........................   Passed   44.06 sec
      Start 1732: test_gdasapp_atm_jjob_var_run
47/52 Test #1732: test_gdasapp_atm_jjob_var_run .........................   Passed  106.12 sec
      Start 1733: test_gdasapp_atm_jjob_var_final
48/52 Test #1733: test_gdasapp_atm_jjob_var_final .......................   Passed   42.14 sec
      Start 1734: test_gdasapp_atm_jjob_ens_init
49/52 Test #1734: test_gdasapp_atm_jjob_ens_init ........................   Passed   43.86 sec
      Start 1735: test_gdasapp_atm_jjob_ens_run
50/52 Test #1735: test_gdasapp_atm_jjob_ens_run .........................   Passed  266.13 sec
      Start 1736: test_gdasapp_atm_jjob_ens_final
51/52 Test #1736: test_gdasapp_atm_jjob_ens_final .......................   Passed   42.14 sec
      Start 1737: test_gdasapp_aero_gen_3dvar_yaml
52/52 Test #1737: test_gdasapp_aero_gen_3dvar_yaml ......................   Passed    0.36 sec

100% tests passed, 0 tests failed out of 52

Label Time Summary:
gdas-utils    =   9.91 sec*proc (9 tests)
script        =   9.91 sec*proc (9 tests)

Total Test time (real) = 1104.51 sec

[CoryMartin-NOAA]

Looks good, nice!

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: orion

Start: Fri Feb 23 14:04:26 CST 2024 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Fri Feb 23 14:55:57 CST 2024
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Fri Feb 23 15:10:52 CST 2024
Tests: 88% tests passed, 6 tests failed out of 52
	1731 - test_gdasapp_atm_jjob_var_init (Failed)
	1732 - test_gdasapp_atm_jjob_var_run (Failed)
	1733 - test_gdasapp_atm_jjob_var_final (Failed)
	1734 - test_gdasapp_atm_jjob_ens_init (Failed)
	1735 - test_gdasapp_atm_jjob_ens_run (Failed)
	1736 - test_gdasapp_atm_jjob_ens_final (Failed)
Tests: see output at /work2/noaa/stmp/cmartin/CI/GDASApp/workflow/PR/933/global-workflow/sorc/gdas.cd/build/log.ctest

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Fri Feb 23 20:11:27 UTC 2024 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Fri Feb 23 21:02:49 UTC 2024
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Fri Feb 23 21:15:44 UTC 2024
Tests: 88% tests passed, 6 tests failed out of 52
	1731 - test_gdasapp_atm_jjob_var_init (Failed)
	1732 - test_gdasapp_atm_jjob_var_run (Failed)
	1733 - test_gdasapp_atm_jjob_var_final (Failed)
	1734 - test_gdasapp_atm_jjob_ens_init (Failed)
	1735 - test_gdasapp_atm_jjob_ens_run (Failed)
	1736 - test_gdasapp_atm_jjob_ens_final (Failed)
Tests: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/933/global-workflow/sorc/gdas.cd/build/log.ctest

[CoryMartin-NOAA]

I believe these are expected to fail because we do not yet have the ability to check out a specific branch of global-workflow in the CI testing. @guillaumevernieres mentioned this previously to me, it is something we should implement "soon"

[RussTreadon-NOAA]

Yes, the Hera and Orion GW-RT failures are expected.

ctests test_gdasapp_atm_jjob_var_init and test_gdasapp_atm_jjob_var_init use g-w python scripts to stage data. The g-w develop python scripts use .nc4 for the radiance bias correction files. This GDASApp PR changes the suffix to .nc. Hence a mismatch and failure. The run jobs fail because the init job did not stage all the required input. The final jobs fail because expected output from the run jobs is not present.

g-w PR #2336 contains updated python driver scripts. As shown above all ctests pass when the changes in this GDASApp PR are built with the changes from g-w PR #2336.

Once GDASApp PR #933 is merged into develop, the pointer for the gdas.cd submodule in g-w PR #2336 will updated to the GDASApp hash conicident with or subsequent to GDASApp PR #933.

[RussTreadon-NOAA]

@CoryMartin-NOAA and @guillaumevernieres , if you are OK with this PR, I'll merge it into develop and update the GDASApp hash in g-w PR #2336. I can then work with the g-w team to get PR #2336 into develop. This will restore Passed for the GDASApp GW-RT JEDI atm and ens ctests.

[CoryMartin-NOAA]

Works for me!

[RussTreadon-NOAA]

Thanks @CoryMartin-NOAA . I'll move forward with the merge into develop.