Enhancement to handle hydrometeors in the GSI and EnKF I/O interfaces by emilyhcliu · Pull Request #499 · NOAA-EMC/GSI

emilyhcliu · 2022-11-22T03:10:52Z

The enhancements made to the GSI/EnKF IO interfaces are the following:
(1) For deterministic I/O: add handling of precipitation hydrometeor in write_gfsncatm (write_atm_) -- write out analysis.
(2) For enkf I/O: add the handling of precipitation hydrometeors to parallel netcdf I.O --- preparation of next implementation

Fixes #498
Depends on #526

RussTreadon-NOAA

Why is file netcdfgfs_io.f90.orig part of this PR? It appears this file was committed by accident. Please remove it.

emilyhcliu · 2022-11-22T21:10:06Z

@RussTreadon-NOAA Extra file removed.

RussTreadon-NOAA · 2022-12-07T14:17:26Z

@emilyhcliu , in order to move forward with this PR we need the following:

include in Enhancement to handle hydrometeors in the GSI and EnKF I/O interfaces #498 regression test results documenting if and how the proposed changes alter existing gsi.x and enkf.x behavior
document via test(s) that the proposed changes achieve the stated goal of Enhancement to handle hydrometeors in the GSI and EnKF I/O interfaces #498. Along this line you should add a new regression test or augment an existing regression test to ensure that subsequent PRs do not break the proposed changes in this PR.
document via test(s) the wall time and memory usage impact of the proposed changes. Doing so is important since this PR increases IO.
identify two subject matter experts to review this PR. I will also review the changes as the Handling Reviewer (see GSI Code Management Policy

emilyhcliu · 2022-12-07T15:58:09Z

@RussTreadon-NOAA Got your points. I will work on them.

emilyhcliu · 2022-12-07T22:23:18Z

Regression test (GSI develop vs. feature/allsky) is done.
All 19 tests passed reproducibility test.
The global_c96_cv3aero test failed the maximum allowable threshold time test
The global_fv3_4denvar_c192 test failed the maximym allowable hardware memory

 1/19 Test  #2: [=[global_T62_ozonly]=] ..........   Passed  724.07 sec
 2/19 Test #19: [=[global_enkf_T62]=] ............   Passed  1687.30 sec
 3/19 Test #11: [=[nmm_netcdf]=] .................   Passed  2223.94 sec
 4/19 Test  #3: [=[global_4dvar_T62]=] ...........   Passed  2227.36 sec
 5/19 Test  #8: [=[arw_netcdf]=] .................   Passed  2284.07 sec
 6/19 Test  #9: [=[arw_binary]=] .................   Passed  2284.57 sec
 7/19 Test #16: [=[netcdf_fv3_regional]=] ........   Passed  2403.78 sec
 8/19 Test  #4: [=[global_4denvar_T126]=] ........   Passed  3186.31 sec
 9/19 Test #13: [=[hwrf_nmm_d2]=] ................   Passed  3971.79 sec
10/19 Test #14: [=[hwrf_nmm_d3]=] ................   Passed  4040.53 sec
11/19 Test #18: [=[global_C96_fv3aerorad]=] ......   Passed  4388.42 sec
12/19 Test #17: [=[global_C96_fv3aero]=] .........***Failed  23483.31 sec (exceed allowable threshold time)
13/19 Test #10: [=[nmm_binary]=] .................   Passed  24093.55 sec
14/19 Test #15: [=[rtma]=] .......................   Passed  24389.57 sec
15/19 Test #12: [=[nmmb_nems_4denvar]=] ..........   Passed  24618.15 sec
16/19 Test  #7: [=[global_lanczos_T62]=] .........   Passed  31167.68 sec
17/19 Test  #6: [=[global_fv3_4denvar_C192]=] ....***Failed  31288.06 sec (exceed allowable hardware memory limit)
18/19 Test  #5: [=[global_fv3_4denvar_T126]=] ....   Passed  31348.51 sec
19/19 Test  #1: [=[global_T62]=] .................   Passed  33147.93 sec

The global_fv3_3denvar_C192 failed the hardware memory limit test:

The memory for global_fv3_4denvar_C192_loproc_updat is 3225364 KBs.  This has exceeded maximum allowable hardware memory limit of 2516582 KBs,
resulting in Failure maxmem of the regression test.

The global_C96_fv3aero failed the allowable threshold time test:

The runtime for global_C96_fv3aero_loproc_updat is 71.750037 seconds.  
This has exceeded maximum allowable threshold time of 58.891738 seconds,
resulting in Failure time-thresh of the regression test.
The runtime for global_C96_fv3aero_hiproc_updat is 32.796763 seconds.  This has exceeded maximum allowable threshold time of 27.241009 seconds,
resulting in Failure of timethresh2 the regression test.

emilyhcliu · 2022-12-08T15:45:34Z

Working on following:
(1) updating regression tests.
(2) documenting the change in GSI execution walltime and memory usage.

RussTreadon-NOAA

Only minor comments. Looks good.

RussTreadon-NOAA · 2022-12-19T17:55:39Z

@emilyhcliu , for this PR to move forward the following needs to be done:

respond to reviewer comments
document wall time and memory impact of additional i/o
if you want to ensure the new functionality is not broken in future updates, the existing global regression test(s) need to be updated.
update your branch to the head of develop

emilyhcliu · 2022-12-20T00:50:26Z

@emilyhcliu , for this PR to move forward the following needs to be done:

respond to reviewer comments

document wall time and memory impact of additional i/o

if you want to ensure the new functionality is not broken in future updates, the existing global regression test(s) need to be updated.

update your branch to the head of develop

I completed the first and the last bullets.
I will work on documenting the time/memory change due to i/o change and global regression tests.

RussTreadon-NOAA

Thank you @emilyhcliu for addressing my code comments.

RussTreadon-NOAA · 2023-01-23T21:38:21Z

@emilyhcliu, just a quick check on the status of this PR. The GSI Review team meets in a few weeks (2/13).

emilyhcliu · 2023-01-23T21:39:25Z

@RussTreadon-NOAA I will complete this within a few days this week.

RussTreadon-NOAA · 2023-05-19T13:18:31Z

+     if(qs_ind > 0) then
+       call read_vardata(dset, 'snmr', ug3d, ncstart=ncstart, nccount=nccount, errcode=iret)
+       if (iret /= 0) then
+           print *,'error reading snmr'


Same error message comment as above.

RussTreadon-NOAA · 2023-05-19T13:19:02Z

+        if (imp_physics == 11) then
+           call read_vardata(dset, 'icmr', vg3d, ncstart=ncstart, nccount=nccount, errcode=iret)
+           if (iret /= 0) then
+              print *,'error reading icmr'


Same error message comment as above.

RussTreadon-NOAA

Need to improve error messages which terminate program execution (NCO requirement). Also a follow on question to Andrew's regarding other instances of use_full_hydro.

RussTreadon-NOAA

Thank you @emilyhcliu for addressing my comments.

RussTreadon-NOAA · 2023-05-19T17:46:21Z

WCOSS2 ctests
Refreshed working copy of emilyhcliu:feature/allsky on Cactus, rebuilt, and reran ctests with the following results

russ.treadon@clogin03:/lfs/h2/emc/da/noscrub/russ.treadon/git/gsi/pr499/build> ctest -j 9
Test project /lfs/h2/emc/da/noscrub/russ.treadon/git/gsi/pr499/build
    Start 1: global_3dvar
    Start 2: global_4dvar
    Start 3: global_4denvar
    Start 4: hwrf_nmm_d2
    Start 5: hwrf_nmm_d3
    Start 6: rtma
    Start 7: rrfs_3denvar_glbens
    Start 8: netcdf_fv3_regional
    Start 9: global_enkf
1/9 Test #8: netcdf_fv3_regional ..............***Failed  543.57 sec
2/9 Test #5: hwrf_nmm_d3 ......................   Passed  553.14 sec
3/9 Test #7: rrfs_3denvar_glbens ..............   Passed  605.27 sec
4/9 Test #4: hwrf_nmm_d2 ......................   Passed  606.73 sec
5/9 Test #9: global_enkf ......................   Passed  609.48 sec
6/9 Test #6: rtma .............................   Passed  1088.08 sec
7/9 Test #3: global_4denvar ...................***Failed  1382.31 sec
8/9 Test #1: global_3dvar .....................   Passed  1444.71 sec
9/9 Test #2: global_4dvar .....................   Passed  1502.15 sec

78% tests passed, 2 tests failed out of 9

Total Test time (real) = 1502.18 sec

The following tests FAILED:
          3 - global_4denvar (Failed)
          8 - netcdf_fv3_regional (Failed)
Errors while running CTest

The netcdf_fv3_regional is a known problem. Non-reproducibility between loproc and hiproc tests entered develop with PR #527. The source for the non-reproducibility is actively being investigated.

The global_4denvar failure is due to the scalability test

The case has Failed the scalability test.
The slope for the update (35.051800 seconds per node) is less than that for the control (44.967220 seconds per node).

A check of the update and control wall times for this test show acceptable values

global_4denvar_hiproc_contrl/stdout:The total amount of wall time                        = 251.990547
global_4denvar_hiproc_updat/stdout:The total amount of wall time                        = 268.302506
global_4denvar_loproc_contrl/stdout:The total amount of wall time                        = 296.957767
global_4denvar_loproc_updat/stdout:The total amount of wall time                        = 296.343946

This PR only affects enkf.x The global_enkf test Passed.

RussTreadon-NOAA · 2023-05-19T17:49:02Z

@emilyhcliu , we need approval from at least two peer reviewers in order to move this PR forward.

CoryMartin-NOAA

I think changes look good here now that they are limited to EnKF I/O only

RussTreadon-NOAA

Ctests run on WCOSS2 from current head of hwrf_nmm_d2_hiproc_updat. global_enkf passed. Given two peer reviewer approvals, move forward with merge into develop.

…C#499) The enhancements made to the EnKF IO interface are the following: (1) For enkf I/O: add the handling of precipitation hydrometeors to parallel netcdf I.O --- preparation of next implementation

emilyhcliu added 2 commits November 22, 2022 02:52

Enhance hydrometeor IO for GSI and EnKF

028ebb8

Enhance hydrometeor IO for EnKF

33b69aa

emilyhcliu self-assigned this Nov 22, 2022

emilyhcliu requested review from ADCollard and RussTreadon-NOAA November 22, 2022 03:11

RussTreadon-NOAA reviewed Nov 22, 2022

View reviewed changes

emilyhcliu and others added 2 commits November 22, 2022 12:32

Delete netcdfgfs_io.f90.orig

7d01091

Merge remote-tracking branch 'upstream/develop' into feature/allsky

b90556c

RussTreadon-NOAA self-requested a review December 19, 2022 17:30

RussTreadon-NOAA reviewed Dec 19, 2022

View reviewed changes

Comment thread src/enkf/gridio_gfs.f90 Outdated

RussTreadon-NOAA reviewed Dec 19, 2022

View reviewed changes

Comment thread src/enkf/gridio_gfs.f90 Outdated

RussTreadon-NOAA reviewed Dec 19, 2022

View reviewed changes

Comment thread src/enkf/gridio_gfs.f90 Outdated

RussTreadon-NOAA reviewed Dec 19, 2022

View reviewed changes

Comment thread src/enkf/gridio_gfs.f90 Outdated

RussTreadon-NOAA reviewed Dec 19, 2022

View reviewed changes

Comment thread src/gsi/netcdfgfs_io.f90 Outdated

RussTreadon-NOAA reviewed Dec 19, 2022

View reviewed changes

emilyhcliu added 5 commits December 20, 2022 00:30

Remove unnecessary comment about future work.

090ee12

Remove extra blank lines

037ba54

Comments for future work removed.

57237dc

Remove obsolete line.

091502a

Merge remote-tracking branch 'upstream/develop' into feature/allsky

b0afd09

RussTreadon-NOAA reviewed Dec 20, 2022

View reviewed changes

ADCollard previously approved these changes May 18, 2023

View reviewed changes