Bug fixes to 3D diagnostic tendencies (based on #12)

[climbfuji]

This PR builds on the original PR #12, but updates the submodule pointer for fv3atm to point to the correct forks and branches for the associated PRs.

Associated PRs:

NOAA-GSL/ccpp-physics#21
NOAA-GSL/fv3atm#19
#16

For regression testing information, see below.

[climbfuji]

@llpcarson @grantfirl @SamuelTrahanNOAA @ligiabernardet please have a look at this draft PR and other draft PRs listed in here. They "massage" the original PRs from @SamuelTrahanNOAA to address some of the issues discussed in NOAA-GSL/ccpp-physics#18.

I can run the GSD suite without ldiag3d/qdiag3d and with ldiag3d/qdiag3d. Both runs give b4b identical results, the ldiag3d/qdiag3d produces a ton of output. I will need to inspect this carefully and see if the numbers make sense. But I think they do, since they are basically all from Sam's original work.

These PRs are still marked as draft PRs, hence consider them as such. But I would appreciate your feedback. Thanks!

[DomHeinzeller]

Note: at present, these PRs change the answer for some tests and for all files after the initialization (including the restart files). This is in REPRO mode.

[DomHeinzeller]

Regression testing on hera.intel using rt_ccpp_gsd.conf

Step 1: Verify against existing code in gsd/develop

These PRs change the answer for some tests and for all files after the initialization (including the restart files). This is in REPRO mode.

These changes are expected, because they result from bug fixes in the suite definition files for the following tests (see commit fed81f9):

fv3_ccpp_thompson_mynn 
fv3_ccpp_gsd           
fv3_ccpp_gsd_coldstart 
fv3_ccpp_gsd_warmstart 
fv3_ccpp_mynn          
fv3_ccpp_gsd_drag_suite
fv3_ccpp_gsd_sar       
fv3_ccpp_gsd_debug     

For all suites, the scheme <scheme>sgscloud_radpost</scheme> was mistakenly run after radiation, too:

...
      <scheme>rrtmg_lw</scheme>
      <scheme>sgscloud_radpre</scheme>
      <scheme>rrtmg_lw_post</scheme>
...

The correct version (which the above commit fixes) is

...
      <scheme>rrtmg_lw</scheme>
      <scheme>sgscloud_radpost</scheme>
      <scheme>rrtmg_lw_post</scheme>
...

Regression tests whose suite definition files are not affected by this bug fix (because they do not use the sgscloud_radd{pre,post} schemes or because the bug had been fixed earlier) give bit-for-bit identical results to a baseline generated by the current gsd/develop branch using rt_ccpp_gsd.conf.

bash-4.2$ grep -e PASS log_hera.intel/*
log_hera.intel/rt_002_fv3_ccpp_gf_thompson_repro.log:Test 002 fv3_ccpp_gf_thompson PASS
log_hera.intel/rt_006_fv3_ccpp_gsd_noah_repro.log:Test 006 fv3_ccpp_gsd_noah PASS
log_hera.intel/rt_007_fv3_ccpp_gsd_mynnsfc_repro.log:Test 007 fv3_ccpp_gsd_mynnsfc PASS
log_hera.intel/rt_008_fv3_ccpp_gsd_noah_mynnsfc_repro.log:Test 008 fv3_ccpp_gsd_noah_mynnsfc PASS
log_hera.intel/rt_009_fv3_ccpp_thompson_repro.log:Test 009 fv3_ccpp_thompson PASS
log_hera.intel/rt_010_fv3_ccpp_thompson_no_aero_repro.log:Test 010 fv3_ccpp_thompson_no_aero PASS
log_hera.intel/rt_011_fv3_ccpp_gf_repro.log:Test 011 fv3_ccpp_gf PASS
log_hera.intel/rt_016_fv3_ccpp_gsd_mynnsfc_debug_prod.log:Test 016 fv3_ccpp_gsd_mynnsfc_debug PASS
log_hera.intel/rt_017_fv3_ccpp_gsd_noah_mynnsfc_debug_prod.log:Test 017 fv3_ccpp_gsd_noah_mynnsfc_debug PASS
log_hera.intel/rt_018_fv3_ccpp_thompson_no_aero_debug_prod.log:Test 018 fv3_ccpp_thompson_no_aero_debug PASS
log_hera.intel/rt_019_fv3_ccpp_gsd_sar_25km_debug_prod.log:Test 019 fv3_ccpp_gsd_sar_25km_debug PASS

rt_ccpp_gsd_verify.log
rt_rt_ccpp_gsd_verify_fail_test.log

Step 2: Create a new baseline and verify against it

Creating a new baseline: all tests pass

rt_ccpp_gsd_create.log

Verifying against it: all tests pass

rt_ccpp_gsd_verify.log

[DomHeinzeller]

@llpcarson @grantfirl @SamuelTrahanNOAA @ligiabernardet please have a look at this draft PR and other draft PRs listed in here. They "massage" the original PRs from @SamuelTrahanNOAA to address some of the issues discussed in NOAA-GSD/ccpp-physics#18.

I can run the GSD suite without ldiag3d/qdiag3d and with ldiag3d/qdiag3d. Both runs give b4b identical results, the ldiag3d/qdiag3d produces a ton of output. I will need to inspect this carefully and see if the numbers make sense. But I think they do, since they are basically all from Sam's original work.

These PRs are still marked as draft PRs, hence consider them as such. But I would appreciate your feedback. Thanks!

All, I marked those PRs as ready for review. Running a final round of tests now, but all looking good thus far.

[DomHeinzeller]

Regression testing on hera.intel using rt.conf:

All tests pass except fv3_ccpp_gsd (in both REPRO and PROD mode). This is expected, because the GSD suite has been updated several times since the official baseline was created.

rt_full.log
rt_full_fail_test.log

[DomHeinzeller]

@llpcarson @grantfirl @SamuelTrahanNOAA @ligiabernardet these PRs are ready to merge.

@tanyasmirnova @joeolson42 @haiqinli @hannahcbarnes please have a look as well and flag anything that looks odd/wrong to you. I'd like to merge this today so that we can update Joe's PRs and focus on getting the MYNN PBL updates in for the HWT experiments. Thanks!

[hannahcbarnes]

@DomHeinzeller I briefly looked over the code and I think it looks alright from a GF perspective.

[ligiabernardet]

Looked at the code only from a high-level perspective and did not spot anything wrong. Thanks for the good work! This will be a helpful capability for developers.

…

On Tue, Apr 21, 2020 at 9:06 AM Hannah C Barnes ***@***.***> wrote: @DomHeinzeller <https://github.com/DomHeinzeller> I briefly looked over the code and I think it looks alright from a GF perspective. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE7WQAS2S2MCNIYO27AKGSTRNWY5DANCNFSM4MKHU4DQ> .

[DomHeinzeller]

Looked at the code only from a high-level perspective and did not spot anything wrong. Thanks for the good work! This will be a helpful capability for developers.
…
On Tue, Apr 21, 2020 at 9:06 AM Hannah C Barnes @.***> wrote: @DomHeinzeller https://github.com/DomHeinzeller I briefly looked over the code and I think it looks alright from a GF perspective. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE7WQAS2S2MCNIYO27AKGSTRNWY5DANCNFSM4MKHU4DQ .

Thanks. I will ignore the missing feedback from others and merge this today so that we can keep moving.