Skip to content

enable atmospheric lgetkf ctests to run in combined and split modes#1262

Merged
RussTreadon-NOAA merged 2 commits into
NOAA-EMC:developfrom
RussTreadon-NOAA:feature/lgetkf_ctests
Aug 26, 2024
Merged

enable atmospheric lgetkf ctests to run in combined and split modes#1262
RussTreadon-NOAA merged 2 commits into
NOAA-EMC:developfrom
RussTreadon-NOAA:feature/lgetkf_ctests

Conversation

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor

This PR generalizes the atmospheric lgetkf ctests to support the following ctests

  • test_gdasapp_atm_jjob_ens_letkf - run lgetkf with observer and solver
  • test_gdasapp_atm_jjob_ens_obs - run lgetkf in observer mode only
  • test_gdasapp_atm_jjob_ens_sol - run lgetkf in solver mode only

These changes are necessitated by changes in g-w PR #2833.

Resolves #1261

@RussTreadon-NOAA RussTreadon-NOAA self-assigned this Aug 23, 2024
@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor Author

@TravisElless-NOAA and @CoryMartin-NOAA , this PR is necessary once g-w PR #2833 is merged into g-w develop.

g-w PR #2833 creates the yaml(s) required by the mode in which lgetkf is run. We do this by checking the value of logical lobsdiag_forenkf in config.base.

When this logical is .true., the hardwired value in g-w, we run eobs & eupd for GSI-based DA. When the logical is .false., we run member eomg, ediag, and eupd for GSI-based DA.

The same approach is used for JEDI-based DA. When the logical is .true. we create yamls to run separate lgetkf observer and solver jobs. When the logical is .false., we create a single yaml to run the lgetkf in combined observer and solver mode.

I retained both the combined observer+solver and separate observe & solver ctests in GDASApp. This is problematic because lobsdiag_forenkf is hardwired to .true. in config.base. This PR creates two config.base to enable all three ctest jobs to successfully run to completion.

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor Author

Hera tests

Install feature/lgetkf_ctests at 80a0297 in g-w feature/ensda_obs at 2ebb823. Run ctests with following results

Test project /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/update/sorc/gdas.cd/build
      Start 1531: test_gdasapp_util_coding_norms
 1/46 Test #1531: test_gdasapp_util_coding_norms ........................   Passed    2.56 sec
      Start 1532: test_gdasapp_util_ioda_example
 2/46 Test #1532: test_gdasapp_util_ioda_example ........................   Passed    1.14 sec
      Start 1533: test_gdasapp_util_prepdata
 3/46 Test #1533: test_gdasapp_util_prepdata ............................   Passed    1.13 sec
      Start 1534: test_gdasapp_util_rads2ioda
 4/46 Test #1534: test_gdasapp_util_rads2ioda ...........................   Passed    0.28 sec
      Start 1535: test_gdasapp_util_ghrsst2ioda
 5/46 Test #1535: test_gdasapp_util_ghrsst2ioda .........................   Passed    0.14 sec
      Start 1536: test_gdasapp_util_rtofstmp
 6/46 Test #1536: test_gdasapp_util_rtofstmp ............................   Passed    1.09 sec
      Start 1537: test_gdasapp_util_rtofssal
 7/46 Test #1537: test_gdasapp_util_rtofssal ............................   Passed    0.79 sec
      Start 1538: test_gdasapp_util_smap2ioda
 8/46 Test #1538: test_gdasapp_util_smap2ioda ...........................   Passed    0.14 sec
      Start 1539: test_gdasapp_util_smos2ioda
 9/46 Test #1539: test_gdasapp_util_smos2ioda ...........................   Passed    0.14 sec
      Start 1540: test_gdasapp_util_viirsaod2ioda
10/46 Test #1540: test_gdasapp_util_viirsaod2ioda .......................   Passed    0.16 sec
      Start 1541: test_gdasapp_util_icecamsr2ioda
11/46 Test #1541: test_gdasapp_util_icecamsr2ioda .......................   Passed    0.14 sec
      Start 1542: test_gdasapp_util_icecmirs2ioda
12/46 Test #1542: test_gdasapp_util_icecmirs2ioda .......................   Passed    0.14 sec
      Start 1880: test_gdasapp_check_python_norms
13/46 Test #1880: test_gdasapp_check_python_norms .......................   Passed    3.36 sec
      Start 1881: test_gdasapp_check_yaml_keys
14/46 Test #1881: test_gdasapp_check_yaml_keys ..........................   Passed    0.25 sec
      Start 1882: test_gdasapp_jedi_increment_to_fv3
15/46 Test #1882: test_gdasapp_jedi_increment_to_fv3 ....................   Passed    0.80 sec
      Start 1883: test_gdasapp_fv3jedi_fv3inc
16/46 Test #1883: test_gdasapp_fv3jedi_fv3inc ...........................   Passed   18.04 sec
      Start 1884: test_gdasapp_soca_nsst_increment_to_mom6
17/46 Test #1884: test_gdasapp_soca_nsst_increment_to_mom6 ..............   Passed    6.69 sec
      Start 1885: test_gdasapp_soca_prep
18/46 Test #1885: test_gdasapp_soca_prep ................................   Passed    4.69 sec
      Start 1886: test_gdasapp_soca_run_clean
19/46 Test #1886: test_gdasapp_soca_run_clean ...........................   Passed    0.04 sec
      Start 1887: test_gdasapp_soca_setup_obsprep
20/46 Test #1887: test_gdasapp_soca_setup_obsprep .......................   Passed   20.95 sec
      Start 1888: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS
21/46 Test #1888: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS ..............   Passed   43.39 sec
      Start 1889: test_gdasapp_soca_JGLOBAL_MARINE_BMAT
22/46 Test #1889: test_gdasapp_soca_JGLOBAL_MARINE_BMAT .................   Passed   74.38 sec
      Start 1890: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP
23/46 Test #1890: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP ....   Passed   42.52 sec
      Start 1891: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN
24/46 Test #1891: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN .....   Passed   42.38 sec
      Start 1892: test_gdasapp_soca_copy_scratch
25/46 Test #1892: test_gdasapp_soca_copy_scratch ........................   Passed    0.49 sec
      Start 1893: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT
26/46 Test #1893: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT ...   Passed   42.49 sec
      Start 1894: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST
27/46 Test #1894: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST ....   Passed   42.38 sec
      Start 1895: test_gdasapp_snow_create_ens
28/46 Test #1895: test_gdasapp_snow_create_ens ..........................   Passed    1.67 sec
      Start 1896: test_gdasapp_snow_imsproc
29/46 Test #1896: test_gdasapp_snow_imsproc .............................   Passed    4.67 sec
      Start 1897: test_gdasapp_snow_apply_jediincr
30/46 Test #1897: test_gdasapp_snow_apply_jediincr ......................   Passed    3.44 sec
      Start 1898: test_gdasapp_snow_letkfoi_snowda
31/46 Test #1898: test_gdasapp_snow_letkfoi_snowda ......................   Passed   13.04 sec
      Start 1899: test_gdasapp_convert_bufr_adpsfc_snow
32/46 Test #1899: test_gdasapp_convert_bufr_adpsfc_snow .................   Passed    3.63 sec
      Start 1900: test_gdasapp_convert_bufr_adpsfc
33/46 Test #1900: test_gdasapp_convert_bufr_adpsfc ......................   Passed    3.98 sec
      Start 1901: test_gdasapp_convert_gsi_satbias
34/46 Test #1901: test_gdasapp_convert_gsi_satbias ......................   Passed    5.42 sec
      Start 1902: test_gdasapp_setup_atm_cycled_exp
35/46 Test #1902: test_gdasapp_setup_atm_cycled_exp .....................   Passed    0.96 sec
      Start 1903: test_gdasapp_atm_jjob_var_init
36/46 Test #1903: test_gdasapp_atm_jjob_var_init ........................   Passed   48.20 sec
      Start 1904: test_gdasapp_atm_jjob_var_run
37/46 Test #1904: test_gdasapp_atm_jjob_var_run .........................   Passed   74.28 sec
      Start 1905: test_gdasapp_atm_jjob_var_inc
38/46 Test #1905: test_gdasapp_atm_jjob_var_inc .........................   Passed   74.29 sec
      Start 1906: test_gdasapp_atm_jjob_var_final
39/46 Test #1906: test_gdasapp_atm_jjob_var_final .......................   Passed   42.28 sec
      Start 1907: test_gdasapp_atm_jjob_ens_init
40/46 Test #1907: test_gdasapp_atm_jjob_ens_init ........................   Passed   48.50 sec
      Start 1908: test_gdasapp_atm_jjob_ens_letkf
41/46 Test #1908: test_gdasapp_atm_jjob_ens_letkf .......................   Passed  458.40 sec
      Start 1909: test_gdasapp_atm_jjob_ens_obs
42/46 Test #1909: test_gdasapp_atm_jjob_ens_obs .........................   Passed   79.07 sec
      Start 1910: test_gdasapp_atm_jjob_ens_sol
43/46 Test #1910: test_gdasapp_atm_jjob_ens_sol .........................   Passed   42.26 sec
      Start 1911: test_gdasapp_atm_jjob_ens_inc
44/46 Test #1911: test_gdasapp_atm_jjob_ens_inc .........................   Passed  106.42 sec
      Start 1912: test_gdasapp_atm_jjob_ens_final
45/46 Test #1912: test_gdasapp_atm_jjob_ens_final .......................   Passed   42.36 sec
      Start 1913: test_gdasapp_aero_gen_3dvar_yaml
46/46 Test #1913: test_gdasapp_aero_gen_3dvar_yaml ......................   Passed    0.71 sec

100% tests passed, 0 tests failed out of 46

Label Time Summary:
gdas-utils    =   7.87 sec*proc (12 tests)
script        =   7.87 sec*proc (12 tests)

Total Test time (real) = 1406.67 sec

Run g-w CI for C96C48_hybatmDA, C96C48_ufs_hybatmDA, C96C48_hybatmaerosnowDA, and C48mx500_3DVarAOWCDA. All jobs in all cycles of each CI configuration successfully run to completion

C96C48_hybatmDA

Hera(hfe05):/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/prgsi$ rocotostat -d prgsi.db -w prgsi.xml -c all -s
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201800        Done    Aug 24 2024 14:10:59    Aug 24 2024 14:30:21
202112210000        Done    Aug 24 2024 14:10:59    Aug 24 2024 16:40:17
202112210600        Done    Aug 24 2024 14:10:59    Aug 24 2024 16:30:22

C96C48_ufs_hybatmDA

Hera(hfe05):/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/prjedi$ rocotostat -d prjedi.db -w prjedi.xml -c all -s
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202402231800        Done    Aug 24 2024 14:11:09    Aug 24 2024 14:30:30
202402240000        Done    Aug 24 2024 14:11:09    Aug 24 2024 17:01:56

C96C48_hybatmaerosnowDA

Hera(hfe05):/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/praero$ rocotostat -d praero.db -w praero.xml                                           -c all -s
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201200        Done    Aug 24 2024 14:10:50    Aug 24 2024 14:35:13
202112201800        Done    Aug 24 2024 14:10:50    Aug 24 2024 15:40:20
202112210000        Done    Aug 24 2024 14:10:50    Aug 24 2024 17:40:18

C48mx500_3DVarAOWCDA

Hera(hfe05):/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/prwcda$ rocotostat -d prwcda.db -w prwcda.xml -c all -s
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103241200        Done    Aug 24 2024 14:11:19    Aug 24 2024 14:30:27
202103241800        Done    Aug 24 2024 14:11:19    Aug 24 2024 15:30:36

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor Author

The sorc/gdas.cd hash in g-w PR #2833 will be updated once this PR is approved and merged into develop.

@TravisElless-NOAA
Copy link
Copy Markdown
Contributor

@RussTreadon-NOAA would there ever be a scenario where these test get flipped around that would require the test_gdasapp_atm_jjob_ens_letkf and test_gdasapp_atm_jjob_ens_sol tests to also have the cp config.base statements as a fail safe?

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor Author

Very good point @TravisElless-NOAA. All three ctests (letkf, obs, sol) now contain a config.base copy to ensure lobsdiag_forenkf is always set correctly for each ctest. Done at 3f6833d

Reran ctest using feature/lgetkf_ctests at 3f6833d. All 46 tests pass. In particular, all local ensemble DA ctests pass

      Start 1907: test_gdasapp_atm_jjob_ens_init
40/46 Test #1907: test_gdasapp_atm_jjob_ens_init ........................   Passed   81.76 sec
      Start 1908: test_gdasapp_atm_jjob_ens_letkf
41/46 Test #1908: test_gdasapp_atm_jjob_ens_letkf .......................   Passed  492.57 sec
      Start 1909: test_gdasapp_atm_jjob_ens_obs
42/46 Test #1909: test_gdasapp_atm_jjob_ens_obs .........................   Passed   76.07 sec
      Start 1910: test_gdasapp_atm_jjob_ens_sol
43/46 Test #1910: test_gdasapp_atm_jjob_ens_sol .........................   Passed   42.54 sec
      Start 1911: test_gdasapp_atm_jjob_ens_inc
44/46 Test #1911: test_gdasapp_atm_jjob_ens_inc .........................   Passed  106.39 sec
      Start 1912: test_gdasapp_atm_jjob_ens_final
45/46 Test #1912: test_gdasapp_atm_jjob_ens_final .......................   Passed   43.28 sec

Copy link
Copy Markdown
Contributor

@TravisElless-NOAA TravisElless-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Things look good to me. Thanks @RussTreadon-NOAA

@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor Author

Thank you @TravisElless-NOAA for your review and approval.

@RussTreadon-NOAA RussTreadon-NOAA merged commit 09594d1 into NOAA-EMC:develop Aug 26, 2024
@RussTreadon-NOAA RussTreadon-NOAA deleted the feature/lgetkf_ctests branch August 26, 2024 20:24
DavidHuber-NOAA added a commit to DavidHuber-NOAA/GDASApp that referenced this pull request Sep 9, 2024
* origin/develop:
  Turn on OMA diagnostics after JEDI minimization (NOAA-EMC#1274)
  Adding AHI and SEVIRI satwnd thinning, enrolling AHI-H9, SEVIRI-M9, SEVIRI-M10 (NOAA-EMC#1263)
  add ctest to create yaml for lgetkf observer ctest (NOAA-EMC#1272)
  add soca fix yaml file for 1deg (NOAA-EMC#1270)
  Fix to marine post and archiving (NOAA-EMC#1266)
  Add JPSSRR sea-ice product to ioda converter (NOAA-EMC#1259)
  enable atmospheric lgetkf ctests to run in combined and split modes (NOAA-EMC#1262)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enable ctests to support single (combined observer & solver) and separate observer & solver ctests

2 participants