gsd/develop: handle blocked data structures in init and finalize routines, remove pset logic, metadata cleanup by climbfuji · Pull Request #23 · NOAA-GSL/ufs-weather-model

climbfuji · 2020-05-27T20:10:19Z

Changes in this PR:

update submodule pointer for fv3atm

Associated PRs:

NOAA-GSL/ccpp-framework#4
NOAA-GSL/ccpp-physics#29
NOAA-GSL/fv3atm#30
#23

For regression testing information, see below.

…nd testing

climbfuji · 2020-05-28T13:50:52Z

Manual testing on macOS with the GNU compilers: results are bit-for-bit identical for 6h forecast runs with the FV3_GSD_v0 suite, except for slightly different values of the purely diagnostic cloud effective radii.

This is because of an actual bugfix (pass the correct optional arguments to mp_thompson_run to trigger the update of the cloud effective radii after the microphysics call). Without this bugfix, the cloud effective radii are updated slightly later in the next time step before the radiation calls, thus the results of the time integration is bit for bit identical.

climbfuji · 2020-05-28T20:33:13Z

Regression testing on hera.intel using rt.conf against official EMC baseline: all tests pass except the expected tests using the GSD suite:

fv3_ccpp_thompson_no_aero_debug
fv3_ccpp_thompson
fv3_ccpp_thompson_no_aero
fv3_ccpp_gsd_debug
fv3_ccpp_thompson_debug
fv3_ccpp_gsd

These tests fail because the physics have been updated and/or new tests have been added (i.e. missing baselines); they all run to completion without crashing.

rt_hera_intel_full_verify.log
rt_hera_intel_full_verify_fail_test.log

climbfuji · 2020-05-28T20:38:13Z

Regression testing on hera.intel using rt_ccpp_gsd.conf against previous results from the gsd/develop codebase:

tests that do not use Thompson MP all pass
tests that use Thompson MP give bit-for-bit identical results except for the purely diagnostic cloud effective radii in REPRO mode (see gsd/develop: handle blocked data structures in init and finalize routines, remove pset logic, metadata cleanup #23 (comment) for an explanation)
tests that use Thompson MP give slightly different results in PROD model; this is because of the changes to the Thompson MP initialization calls, for which the compiler optimization gives slightly different answers

@hannahcbarnes is currently conducting a ten-testcase retrospective run to ascertain that the differences due to the compiler optimization do not have any statistically significant impact on the results.

TO BE COMPLETED

climbfuji · 2020-05-28T20:39:06Z

Regression testing on hera.intel using rt_ccpp_gsd.conf to confirm bit-for-bit identical results in repeated runs (i.e. first create new baseline, then verify against it): both the create and the verify step pass.

rt_hera_intel_ccpp_gsd_create_new_baseline.log
rt_hera_intel_ccpp_gsd_verify_against_new_baseline.log

climbfuji · 2020-05-28T20:52:02Z

@gold2718 @ligiabernardet @llpcarson @grantfirl @mzhangw @JulieSchramm - for your information, this is a fairly significant update of the CCPP framework capabilities to handle blocked data structures in the _init and _finalize routines. These changes needed to be made in gsd/develop, because we have an ideal use case here (initializing Thompson MP), for which we had to use a nasty workaround until now (essentially pass in the entire IPD_Data DDT and then use pointers to be able to use the existing code without too many changes). This workaround needed to be addressed before sending gsd/develop back to master. Since gsd/develop will be merged to master in early June, this capability will be available to everyone soon.

Note that the name of the new metadata attribute active is preliminary and most likely will be changed in a follow-up commit (before merging to master), based on discussions with the other CCPP framework developers. We had previously discussed an attribute called debug_check or check_debug with essentially the same functionality, but just for debugging purposes. Now it turns out that we need this information also for handling blocked data structures and unit conversions (for which we ignored the issue of potentially trying to convert unallocated/invalid data).

Another important change is the removal of the infamous pset logic, which only existed to support the dynamic CCPP build.

Both together are a big step towards transitioning to cap_gen.py, since they allow us to work from both ends.

climbfuji · 2020-05-28T21:06:28Z

Here is an example for handling blocked data structures for mp_thompson_init (in the auto-generated ccpp_FV3_GSD_v0_physics_cap.F90):

   function FV3_GSD_v0_physics_init_cap(GFS_Data,GFS_Control,con_p0,con_rd,con_g,cdata) result(ierr)

...

      if (.true.) then

        allocate(tmpvar1(1:GFS_Control%ncols,1:GFS_Control%levs))
        ib = 1
        do nb=1,GFS_Control%nblks
          tmpvar1(ib:ib+GFS_Control%blksz(nb)-1,:) = GFS_Data(nb)%Statein%qgrs(:,:,GFS_Control%ntqv)
          ib = ib+GFS_Control%blksz(nb)
        end do
      end if

      if (.true.) then

        allocate(tmpvar2(1:GFS_Control%ncols,1:GFS_Control%levs))
        ib = 1
        do nb=1,GFS_Control%nblks
          tmpvar2(ib:ib+GFS_Control%blksz(nb)-1,:) = GFS_Data(nb)%Statein%qgrs(:,:,GFS_Control%ntcw)
          ib = ib+GFS_Control%blksz(nb)
        end do
      end if

      if (.true.) then

        allocate(tmpvar3(1:GFS_Control%ncols,1:GFS_Control%levs))
        ib = 1
        do nb=1,GFS_Control%nblks
          tmpvar3(ib:ib+GFS_Control%blksz(nb)-1,:) = GFS_Data(nb)%Statein%qgrs(:,:,GFS_Control%ntrw)
          ib = ib+GFS_Control%blksz(nb)
        end do
      end if

      if (.true.) then

        allocate(tmpvar4(1:GFS_Control%ncols,1:GFS_Control%levs))
        ib = 1
        do nb=1,GFS_Control%nblks
          tmpvar4(ib:ib+GFS_Control%blksz(nb)-1,:) = GFS_Data(nb)%Statein%qgrs(:,:,GFS_Control%ntiw)
          ib = ib+GFS_Control%blksz(nb)
        end do
      end if

      if (.true.) then

        allocate(tmpvar5(1:GFS_Control%ncols,1:GFS_Control%levs))
        ib = 1
        do nb=1,GFS_Control%nblks
          tmpvar5(ib:ib+GFS_Control%blksz(nb)-1,:) = GFS_Data(nb)%Statein%qgrs(:,:,GFS_Control%ntsw)
          ib = ib+GFS_Control%blksz(nb)
        end do
      end if

      if (.true.) then

        allocate(tmpvar6(1:GFS_Control%ncols,1:GFS_Control%levs))
        ib = 1
        do nb=1,GFS_Control%nblks
          tmpvar6(ib:ib+GFS_Control%blksz(nb)-1,:) = GFS_Data(nb)%Statein%qgrs(:,:,GFS_Control%ntgl)
          ib = ib+GFS_Control%blksz(nb)
        end do
      end if

      if (.true.) then

        allocate(tmpvar7(1:GFS_Control%ncols,1:GFS_Control%levs))
        ib = 1
        do nb=1,GFS_Control%nblks
          tmpvar7(ib:ib+GFS_Control%blksz(nb)-1,:) = GFS_Data(nb)%Statein%qgrs(:,:,GFS_Control%ntinc)
          ib = ib+GFS_Control%blksz(nb)
        end do
      end if

      if (.true.) then

        allocate(tmpvar8(1:GFS_Control%ncols,1:GFS_Control%levs))
        ib = 1
        do nb=1,GFS_Control%nblks
          tmpvar8(ib:ib+GFS_Control%blksz(nb)-1,:) = GFS_Data(nb)%Statein%qgrs(:,:,GFS_Control%ntrnc)
          ib = ib+GFS_Control%blksz(nb)
        end do
      end if

      if ((GFS_Control%ntlnc > 0)) then

        allocate(tmpvar9(1:GFS_Control%ncols,1:GFS_Control%levs))
        ib = 1
        do nb=1,GFS_Control%nblks
          tmpvar9(ib:ib+GFS_Control%blksz(nb)-1,:) = GFS_Data(nb)%Statein%qgrs(:,:,GFS_Control%ntlnc)
          ib = ib+GFS_Control%blksz(nb)
        end do
      end if

      if ((GFS_Control%imp_physics == GFS_Control%imp_physics_thompson .and. GFS_Control%ltaerosol)) then

        allocate(tmpvar10(1:GFS_Control%ncols))
        ib = 1
        do nb=1,GFS_Control%nblks
          tmpvar10(ib:ib+GFS_Control%blksz(nb)-1) = GFS_Data(nb)%Coupling%nwfa2d
          ib = ib+GFS_Control%blksz(nb)
        end do
      end if

      if ((GFS_Control%imp_physics == GFS_Control%imp_physics_thompson .and. GFS_Control%ltaerosol)) then

        allocate(tmpvar11(1:GFS_Control%ncols))
        ib = 1
        do nb=1,GFS_Control%nblks
          tmpvar11(ib:ib+GFS_Control%blksz(nb)-1) = GFS_Data(nb)%Coupling%nifa2d
          ib = ib+GFS_Control%blksz(nb)
        end do
      end if

      if ((GFS_Control%ntwa > 0)) then

        allocate(tmpvar12(1:GFS_Control%ncols,1:GFS_Control%levs))
        ib = 1
        do nb=1,GFS_Control%nblks
          tmpvar12(ib:ib+GFS_Control%blksz(nb)-1,:) = GFS_Data(nb)%Statein%qgrs(:,:,GFS_Control%ntwa)
          ib = ib+GFS_Control%blksz(nb)
        end do
      end if

      if ((GFS_Control%ntia > 0)) then

        allocate(tmpvar13(1:GFS_Control%ncols,1:GFS_Control%levs))
        ib = 1
        do nb=1,GFS_Control%nblks
          tmpvar13(ib:ib+GFS_Control%blksz(nb)-1,:) = GFS_Data(nb)%Statein%qgrs(:,:,GFS_Control%ntia)
          ib = ib+GFS_Control%blksz(nb)
        end do
      end if

      if (.true.) then

        allocate(tmpvar14(1:GFS_Control%ncols,1:GFS_Control%levs))
        ib = 1
        do nb=1,GFS_Control%nblks
          tmpvar14(ib:ib+GFS_Control%blksz(nb)-1,:) = GFS_Data(nb)%Statein%tgrs
          ib = ib+GFS_Control%blksz(nb)
        end do
      end if

      if (.true.) then

        allocate(tmpvar15(1:GFS_Control%ncols,1:GFS_Control%levs))
        ib = 1
        do nb=1,GFS_Control%nblks
          tmpvar15(ib:ib+GFS_Control%blksz(nb)-1,:) = GFS_Data(nb)%Statein%prsl
          ib = ib+GFS_Control%blksz(nb)
        end do
      end if

      if (.true.) then

        allocate(tmpvar16(1:GFS_Control%ncols,1:GFS_Control%levs))
        ib = 1
        do nb=1,GFS_Control%nblks
          tmpvar16(ib:ib+GFS_Control%blksz(nb)-1,:) = GFS_Data(nb)%Statein%phil
          ib = ib+GFS_Control%blksz(nb)
        end do
      end if

      if (.true.) then

        allocate(tmpvar17(1:GFS_Control%ncols))
        ib = 1
        do nb=1,GFS_Control%nblks
          tmpvar17(ib:ib+GFS_Control%blksz(nb)-1) = GFS_Data(nb)%Grid%area
          ib = ib+GFS_Control%blksz(nb)
        end do
      end if

      if (.true.) then

        allocate(tmpvar18(1:GFS_Control%ncols,1:GFS_Control%levs))
        ib = 1
        do nb=1,GFS_Control%nblks
          tmpvar18(ib:ib+GFS_Control%blksz(nb)-1,:) = GFS_Data(nb)%Tbd%phy_f3d(:,:,GFS_Control%nleffr)
          ib = ib+GFS_Control%blksz(nb)
        end do
      end if

      if (.true.) then

        allocate(tmpvar19(1:GFS_Control%ncols,1:GFS_Control%levs))
        ib = 1
        do nb=1,GFS_Control%nblks
          tmpvar19(ib:ib+GFS_Control%blksz(nb)-1,:) = GFS_Data(nb)%Tbd%phy_f3d(:,:,GFS_Control%nieffr)
          ib = ib+GFS_Control%blksz(nb)
        end do
      end if

      if (.true.) then

        allocate(tmpvar20(1:GFS_Control%ncols,1:GFS_Control%levs))
        ib = 1
        do nb=1,GFS_Control%nblks
          tmpvar20(ib:ib+GFS_Control%blksz(nb)-1,:) = GFS_Data(nb)%Tbd%phy_f3d(:,:,GFS_Control%nseffr)
          ib = ib+GFS_Control%blksz(nb)
        end do
      end if

      call mp_thompson_init(ncol=GFS_Control%ncols,nlev=GFS_Control%levs,con_g=con_g,con_rd=con_rd, &
                  restart=GFS_Control%restart,imp_physics=GFS_Control%imp_physics,imp_physics_thompson=GFS_Control%imp_physics_thompson, &
                  spechum=tmpvar1,qc=tmpvar2,qr=tmpvar3,qi=tmpvar4,qs=tmpvar5,qg=tmpvar6, &
                  ni=tmpvar7,nr=tmpvar8,is_aerosol_aware=GFS_Control%ltaerosol,nc=tmpvar9, &
                  nwfa2d=tmpvar10,nifa2d=tmpvar11,nwfa=tmpvar12,nifa=tmpvar13,tgrs=tmpvar14, &
                  prsl=tmpvar15,phil=tmpvar16,area=tmpvar17,re_cloud=tmpvar18,re_ice=tmpvar19, &
                  re_snow=tmpvar20,mpicomm=GFS_Control%communicator,mpirank=GFS_Control%me, &
                  mpiroot=GFS_Control%master,threads=GFS_Control%nthreads,errmsg=cdata%errmsg, &
                  errflg=cdata%errflg)


      if (.true.) then

        ib = 1
        do nb=1,GFS_Control%nblks
          GFS_Data(nb)%Statein%qgrs(:,:,GFS_Control%ntqv) = tmpvar1(ib:ib+GFS_Control%blksz(nb)-1,:)
          ib = ib+GFS_Control%blksz(nb)
        end do
        deallocate(tmpvar1)
      end if

      if (.true.) then

        ib = 1
        do nb=1,GFS_Control%nblks
          GFS_Data(nb)%Statein%qgrs(:,:,GFS_Control%ntcw) = tmpvar2(ib:ib+GFS_Control%blksz(nb)-1,:)
          ib = ib+GFS_Control%blksz(nb)
        end do
        deallocate(tmpvar2)
      end if

      if (.true.) then

        ib = 1
        do nb=1,GFS_Control%nblks
          GFS_Data(nb)%Statein%qgrs(:,:,GFS_Control%ntrw) = tmpvar3(ib:ib+GFS_Control%blksz(nb)-1,:)
          ib = ib+GFS_Control%blksz(nb)
        end do
        deallocate(tmpvar3)
      end if

      if (.true.) then

        ib = 1
        do nb=1,GFS_Control%nblks
          GFS_Data(nb)%Statein%qgrs(:,:,GFS_Control%ntiw) = tmpvar4(ib:ib+GFS_Control%blksz(nb)-1,:)
          ib = ib+GFS_Control%blksz(nb)
        end do
        deallocate(tmpvar4)
      end if

      if (.true.) then

        ib = 1
        do nb=1,GFS_Control%nblks
          GFS_Data(nb)%Statein%qgrs(:,:,GFS_Control%ntsw) = tmpvar5(ib:ib+GFS_Control%blksz(nb)-1,:)
          ib = ib+GFS_Control%blksz(nb)
        end do
        deallocate(tmpvar5)
      end if

      if (.true.) then

        ib = 1
        do nb=1,GFS_Control%nblks
          GFS_Data(nb)%Statein%qgrs(:,:,GFS_Control%ntgl) = tmpvar6(ib:ib+GFS_Control%blksz(nb)-1,:)
          ib = ib+GFS_Control%blksz(nb)
        end do
        deallocate(tmpvar6)
      end if

      if (.true.) then

        ib = 1
        do nb=1,GFS_Control%nblks
          GFS_Data(nb)%Statein%qgrs(:,:,GFS_Control%ntinc) = tmpvar7(ib:ib+GFS_Control%blksz(nb)-1,:)
          ib = ib+GFS_Control%blksz(nb)
        end do
        deallocate(tmpvar7)
      end if

      if (.true.) then

        ib = 1
        do nb=1,GFS_Control%nblks
          GFS_Data(nb)%Statein%qgrs(:,:,GFS_Control%ntrnc) = tmpvar8(ib:ib+GFS_Control%blksz(nb)-1,:)
          ib = ib+GFS_Control%blksz(nb)
        end do
        deallocate(tmpvar8)
      end if

      if ((GFS_Control%ntlnc > 0)) then

        ib = 1
        do nb=1,GFS_Control%nblks
          GFS_Data(nb)%Statein%qgrs(:,:,GFS_Control%ntlnc) = tmpvar9(ib:ib+GFS_Control%blksz(nb)-1,:)
          ib = ib+GFS_Control%blksz(nb)
        end do
        deallocate(tmpvar9)
      end if

      if ((GFS_Control%imp_physics == GFS_Control%imp_physics_thompson .and. GFS_Control%ltaerosol)) then

        ib = 1
        do nb=1,GFS_Control%nblks
          GFS_Data(nb)%Coupling%nwfa2d = tmpvar10(ib:ib+GFS_Control%blksz(nb)-1)
          ib = ib+GFS_Control%blksz(nb)
        end do
        deallocate(tmpvar10)
      end if

      if ((GFS_Control%imp_physics == GFS_Control%imp_physics_thompson .and. GFS_Control%ltaerosol)) then

        ib = 1
        do nb=1,GFS_Control%nblks
          GFS_Data(nb)%Coupling%nifa2d = tmpvar11(ib:ib+GFS_Control%blksz(nb)-1)
          ib = ib+GFS_Control%blksz(nb)
        end do
        deallocate(tmpvar11)
      end if

      if ((GFS_Control%ntwa > 0)) then

        ib = 1
        do nb=1,GFS_Control%nblks
          GFS_Data(nb)%Statein%qgrs(:,:,GFS_Control%ntwa) = tmpvar12(ib:ib+GFS_Control%blksz(nb)-1,:)
          ib = ib+GFS_Control%blksz(nb)
        end do
        deallocate(tmpvar12)
      end if

      if ((GFS_Control%ntia > 0)) then

        ib = 1
        do nb=1,GFS_Control%nblks
          GFS_Data(nb)%Statein%qgrs(:,:,GFS_Control%ntia) = tmpvar13(ib:ib+GFS_Control%blksz(nb)-1,:)
          ib = ib+GFS_Control%blksz(nb)
        end do
        deallocate(tmpvar13)
      end if

      if (.true.) then

        deallocate(tmpvar14)
      end if

      if (.true.) then

        deallocate(tmpvar15)
      end if

      if (.true.) then

        deallocate(tmpvar16)
      end if

      if (.true.) then

        deallocate(tmpvar17)
      end if

      if (.true.) then

        ib = 1
        do nb=1,GFS_Control%nblks
          GFS_Data(nb)%Tbd%phy_f3d(:,:,GFS_Control%nleffr) = tmpvar18(ib:ib+GFS_Control%blksz(nb)-1,:)
          ib = ib+GFS_Control%blksz(nb)
        end do
        deallocate(tmpvar18)
      end if

      if (.true.) then

        ib = 1
        do nb=1,GFS_Control%nblks
          GFS_Data(nb)%Tbd%phy_f3d(:,:,GFS_Control%nieffr) = tmpvar19(ib:ib+GFS_Control%blksz(nb)-1,:)
          ib = ib+GFS_Control%blksz(nb)
        end do
        deallocate(tmpvar19)
      end if

      if (.true.) then

        ib = 1
        do nb=1,GFS_Control%nblks
          GFS_Data(nb)%Tbd%phy_f3d(:,:,GFS_Control%nseffr) = tmpvar20(ib:ib+GFS_Control%blksz(nb)-1,:)
          ib = ib+GFS_Control%blksz(nb)
        end do
        deallocate(tmpvar20)
      end if

      if (cdata%errflg/=0) then
        cdata%errmsg = "An error occured in mp_thompson_init: " // trim(cdata%errmsg)
        ierr=cdata%errflg
        return
      end if

...

   end function FV3_GSD_v0_physics_init_cap

climbfuji · 2020-05-28T21:07:55Z

@dustinswales for your information, the fv3atm and ccpp-physics PRs contain some bugfixes for RRTMGP that I discovered when hardening some of the error checking functionality in ccpp_prebuild.py.

climbfuji · 2020-05-28T21:41:25Z

@tanyasmirnova @joeolson42 @haiqinli @hannahcbarnes these PRs are ready for review, and once Hannah completed the retrospective runs, hopefully also ready to merge. Please have a good look. Most of the changes in the ccpp-physics PR are about fixing inconsistent use of horizontal_dimension and horizontal_loop_extent in the same routine.

tanyasmirnova · 2020-05-29T18:47:42Z

Dom,
I ran a case for 20 Nov 2019 and found the differences in the 21-h fcst from your branch and top of gsd/develop branch (checkout out today) that I did not expect. Below is the plot for 2-m T difference:

The test directory is /scratch1/BMC/gsd-fv3-dev/smirnova/fv3_rt/MANUAL_TEST_C768_20191120_gsdnoah_dom28may20

Did you expect to see this kind of differences?
Thanks,
Tanya

joeolson42 · 2020-05-29T19:31:13Z

Yes, some of that cooling is due to the reduced sgs ice clouds. Dave Turner has recently showed me some results from DOE's SGP and Alaska site, which shows that we have loe ice water path, so I will retract that modification. That alone will help, but this version of the MYNN has the same low diffusivity as in the current HRRR, which I also think isn't suitable for the global applications. I'll probably have to increase the mixing lengths back to their former global values.

…

-joe

On Fri, May 29, 2020 at 12:48 PM tanyasmirnova ***@***.***> wrote: Dom, I ran a case for 20 Nov 2019 and found the differences in the 21-h fcst from your branch and top of gsd/develop branch (checkout out today) that I did not expect. Below is the plot for 2-m T difference: [image: Screen Shot 2020-05-29 at 12 42 39 PM] <https://user-images.githubusercontent.com/38667904/83294270-473e5100-a1aa-11ea-9e2a-2aeb604a212c.png> The test directory is /scratch1/BMC/gsd-fv3-dev/smirnova/fv3_rt/MANUAL_TEST_C768_20191120_gsdnoah_dom28may20 Did you expect to see this kind of differences? Thanks, Tanya — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#23 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADLRR3RS7W6EEZLPSYHGNUTRT77NXANCNFSM4NMOZGUA> .

-- Joseph Olson Meteorologist Environmental Prediction Advancement Division Global Systems Laboratory NOAA-Earth System Research Labs Boulder, Colorado

tanyasmirnova · 2020-05-29T20:24:21Z

I saw that this PR has the following change for the MYNN PBL:
physics/module_bl_mynn.F90: fix compiler warning
Therefore, I thought that the results should be quite close.

Also, In Hannah's ppts we see a significant cold bias near the surface in the polar areas. Why do we want to make it even cooler?

climbfuji · 2020-05-29T21:12:24Z

Tanya is right, I do not expect any statistically significant differences in the runs. There were no science changes, just removing the workaround for the Thompson MP scheme and a bugfix that prevented updating the cloud effective radii as part of the Thompson MP call. But this only has an effect for the diagnostic output, because the effective radii are calculated again before radiation. We could get rid of the redundant calculation in Thompson MP, actually.

Tanya, can you recompile the code with REPRO=Y for both the head of gsd/develop and my branch and redo the forecasts, please?

climbfuji · 2020-05-29T21:22:24Z

Just to clarify, I expect the runs to be b4b identical in REPRO mode. If this is the case, then there is nothing we can do other than accept the changes and fix the actual cause as Joe described above. @tanyasmirnova can you please clarify which suite definition file you used, and which compiler options?

joeolson42 · 2020-05-29T21:33:10Z

It's not clear to me which code versions Tanya is comparing.

…

-joe

On Fri, May 29, 2020 at 3:22 PM Dom Heinzeller ***@***.***> wrote: Just to clarify, I expect the runs to be b4b identical in REPRO mode. If this is the case, then there is nothing we can do other than accept the changes and fix the actual cause as Joe described above. @tanyasmirnova <https://github.com/tanyasmirnova> can you please clarify which suite definition file you used, and which compiler options? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#23 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADLRR3XEBNNG7KJAFUX3ZTLRUARR3ANCNFSM4NMOZGUA> .

-- Joseph Olson Meteorologist Environmental Prediction Advancement Division Global Systems Laboratory NOAA-Earth System Research Labs Boulder, Colorado

tanyasmirnova · 2020-05-29T21:34:18Z

Dom,
I ran a FV3_GSD_noah suite.
The compiling options used are:
Elapsed time 949 seconds. Compiling CCPP=Y REPRO=Y STATIC=Y SUITES=FV3_GFS_v16beta,FV3_GSD_v0,FV3_GSD_v0_mynnsfc,FV3_GSD_noah,FV3_GSD_noah_mynnsfc finished

The run directories:
MANUAL_TEST_C768_20191120_gsdnoah_dom28may20
compiled in
/scratch1/BMC/gsd-fv3-dev/smirnova/ccpp_fv3/UFS_dom_28may20_build
and
MANUAL_TEST_C768_20191120_gsdnoah_29may20
compiled in
/scratch1/BMC/gsd-fv3-dev/smirnova/ccpp_fv3/UFS_29may20

tanyasmirnova · 2020-05-29T21:35:41Z

Joe,
I compare top of gsd./develop branch and
Dom's code:
git clone -b gsd_develop_blocked_data_structures --recursive https://github.com/climbfuji/ufs-weather-model

climbfuji · 2020-05-29T21:38:08Z

Joe,
I compare top of gsd./develop branch and
Dom's code:
git clone -b gsd_develop_blocked_data_structures --recursive https://github.com/climbfuji/ufs-weather-model

Very confusing. I will need to check your code and run directories, as well as my PR once more.

joeolson42 · 2020-05-29T21:40:07Z

OK, then I agree, this is a very confusing result...

…

On Fri, May 29, 2020 at 3:38 PM Dom Heinzeller ***@***.***> wrote: Joe, I compare top of gsd./develop branch and Dom's code: git clone -b gsd_develop_blocked_data_structures --recursive https://github.com/climbfuji/ufs-weather-model Very confusing. I will need to check your code and run directories, as well as my PR once more. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#23 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADLRR3XBY4ZY6FW5O5CGCPLRUATM3ANCNFSM4NMOZGUA> .

-- Joseph Olson Meteorologist Environmental Prediction Advancement Division Global Systems Laboratory NOAA-Earth System Research Labs Boulder, Colorado

…er required

climbfuji · 2020-05-29T23:24:07Z

I just ran the following three regression tests using the head of gsd/develop and the code in these PRs (and yes, both are identical with the source codes that Tanya used):

#################################################################################################################################################################################
# CCPP STATIC tests                                                                                                                                                             #
#################################################################################################################################################################################
# Compile with CCPP - static mode
COMPILE | CCPP=Y REPRO=Y SUITES=FV3_GFS_v15_thompson_mynn,FV3_GFS_v15_gf_thompson,FV3_GSD_v0,FV3_GSD_noah,FV3_GFS_v15_gf           | standard    | hera.intel     | fv3         |
COMPILE | CCPP=Y REPRO=Y SUITES=FV3_GFS_v15_thompson_mynn,FV3_GFS_v15_gf_thompson,FV3_GSD_v0,FV3_GSD_noah,FV3_GFS_v15_gf           | standard    | cheyenne.intel | fv3         |
COMPILE | CCPP=Y REPRO=Y SUITES=FV3_GFS_v15_thompson_mynn,FV3_GFS_v15_gf_thompson,FV3_GSD_v0,FV3_GSD_noah,FV3_GFS_v15_gf           | standard    | cheyenne.gnu   | fv3         |
# Run tests
RUN     | fv3_ccpp_gf_thompson                                                                                                     | standard    |                | fv3         |
RUN     | fv3_ccpp_gsd                                                                                                             | standard    |                | fv3         |
RUN     | fv3_ccpp_gsd_noah                                                                                                        | standard    |                | fv3         |
RUN     | fv3_ccpp_gf                                                                                                              | standard    |                | fv3         |

The results are b4b identical in all checksums and all files, except the purely diagnostic cloud effective radii in RESTART/phy_data.tile{1-6}.nc, as explained earlier.

I will do prod runs next and see what kind of differences I get in 24h integrations, compared to each other and compared to the retro runs.

climbfuji · 2020-05-30T02:30:17Z

Update: same story for the PROD runs - only difference is in RESTART/phy_data.tile{1-6}.nc for the effective radii as diagnostic output, otherwise b4b identical. Next? Copy Tanya's run directory, check what is different and see what my executables do. Maybe not today anymore.

climbfuji · 2020-05-31T21:45:59Z

Next update: despite these identical results, I can reproduce the differences that @tanyasmirnova found for her regional c768 test case in REPRO mode.

climbfuji · 2020-06-01T13:30:08Z

@tanyasmirnova @joeolson42 @hannahcbarnes I got something. When I remove the effr_in = .true. statement in Tanya's input.nml, then I get bit-for-bit identical results between the original code (head of gsd/develop) and my branch. At least I know where to look now.

climbfuji · 2020-06-01T15:05:46Z

Here is a screenshot of the original code (head of gsd/develop) with effr_in = .false. on the left (REPRO mode, for which this PR gives b4b identical results), compared to the same code with effr_in = .true. on the right; left minus right in the middle. This is for @tanyasmirnova's C768 run.

Clearly, the colder temperatures in the arctic regions come from turning off using the effective radii in the RRTMG pre/clouds code for Thompson MP.

With respect to the cold bias introduced by these PRs: this is because I fixed a bug in the CCPP prebuild config that prevented the effective radii being calculated in Thompson MP (i.e. after the actual microphysics calls). We should turn this off correctly and document it, because it is not needed if those radii are calculated before radiation anyway (and apparently in a slightly different way - to be discussed in today's telecon). @tanyasmirnova @hannahcbarnes @joeolson42

…led model (ufs-community#254) * Implements a bugfix for the mapping of ifrac for both nems_orig and nems_frac in CMEPS: issue (#17)[NOAA-EMC/CMEPS#17] * Adds field-packing for performance improvements in CMEPS: issue (#23)[NOAA-EMC/CMEPS#23] * Uses module file fv3_debug when DEBUG=Y is specified and fv3_debug is available * Shortens the nems.configure file name used with the coupled model

Update .gitmodules and submodule pointer for fv3atm for code review a…

0aae4b6

…nd testing

climbfuji force-pushed the gsd_develop_blocked_data_structures branch 2 times, most recently from 69cfe71 to c6806da Compare May 28, 2020 20:16

climbfuji marked this pull request as ready for review May 28, 2020 20:16

climbfuji requested a review from DomHeinzeller as a code owner May 28, 2020 20:16

tests/rt_ccpp_gsd.conf: remove STATIC=Y from compile options, no long…

059912c

…er required

climbfuji force-pushed the gsd_develop_blocked_data_structures branch from c6806da to 059912c Compare May 29, 2020 22:47

Revert change to .gitmodules and update submodule pointer for fv3atm

72cd56a

DomHeinzeller approved these changes Jun 1, 2020

View reviewed changes

DomHeinzeller merged commit 7aa630e into NOAA-GSL:gsd/develop Jun 1, 2020

Conversation

climbfuji commented May 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

climbfuji commented May 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

climbfuji commented May 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

climbfuji commented May 28, 2020

Uh oh!

climbfuji commented May 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

climbfuji commented May 28, 2020

Uh oh!

climbfuji commented May 28, 2020

Uh oh!

climbfuji commented May 28, 2020

Uh oh!

climbfuji commented May 28, 2020

Uh oh!

tanyasmirnova commented May 29, 2020

Uh oh!

joeolson42 commented May 29, 2020 via email

Uh oh!

tanyasmirnova commented May 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

climbfuji commented May 29, 2020

Uh oh!

climbfuji commented May 29, 2020

Uh oh!

joeolson42 commented May 29, 2020 via email

Uh oh!

tanyasmirnova commented May 29, 2020

Uh oh!

tanyasmirnova commented May 29, 2020

Uh oh!

climbfuji commented May 29, 2020

Uh oh!

joeolson42 commented May 29, 2020 via email

Uh oh!

climbfuji commented May 29, 2020

Uh oh!

climbfuji commented May 30, 2020

Uh oh!

climbfuji commented May 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

climbfuji commented Jun 1, 2020

Uh oh!

climbfuji commented Jun 1, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

climbfuji commented May 27, 2020 •

edited

Loading

climbfuji commented May 28, 2020 •

edited

Loading

climbfuji commented May 28, 2020 •

edited

Loading

climbfuji commented May 28, 2020 •

edited

Loading

tanyasmirnova commented May 29, 2020 •

edited

Loading

climbfuji commented May 31, 2020 •

edited

Loading