Skip to content

Regain bit-for-bit identical results between IPD and CCPP for coupled model runs #86

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
junwang-noaa merged 13 commits into from
Mar 27, 2020
Merged

Regain bit-for-bit identical results between IPD and CCPP for coupled model runs #86

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
a47e0ce
Merge pull request #1 from NOAA-EMC/develop
pjpegion Dec 6, 2019
295f1da
Merge pull request #2 from NOAA-EMC/develop
pjpegion Jan 22, 2020
ae32865
Merge pull request #3 from NOAA-EMC/develop
pjpegion Mar 19, 2020
18910aa
make rain/snow tendency consistent with accumulated rain/snow
pjpegion Mar 20, 2020
589ecfd
put drain_cpl and dsnow_cpl in proper container
pjpegion Mar 20, 2020
c436789
Updates of IPD and CCPP code to regain bit-for-bit identical results …
climbfuji Mar 24, 2020
ea84cb5
Update .gitmodules and submodule pointer for ccpp-physics for code re…
climbfuji Mar 24, 2020
fcba0e9
gfsphysics/GFS_layer/GFS_physics_driver.F90: need to initialize local…
climbfuji Mar 24, 2020
9528d48
Merge branch 'coupled_stochy_fixes' of https://github.com/pjpegion/fv…
climbfuji Mar 24, 2020
6f4cb56
Bugfix in gfsphysics/GFS_layer/GFS_typedefs.F90, allocate Tbd%drain_c…
climbfuji Mar 24, 2020
bcda876
Remove local/interstitial variables for seaice coupling, add suite de…
climbfuji Mar 25, 2020
1c8ce05
Rename S2S suite suite_FV3_GFS_2017_coupled_satmedmf.xml to suite_FV3…
climbfuji Mar 26, 2020
d4380df
Revert change to .gitmodules and update submodule pointer for ccpp-ph…
climbfuji Mar 27, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ccpp/physics
Open in desktop
Submodule physics updated 13 files
+9 −16 physics/GFS_MP_generic.F90
+12 −9 physics/GFS_PBL_generic.F90
+8 −8 physics/GFS_PBL_generic.meta
+21 −3 physics/GFS_debug.F90
+2 −2 physics/GFS_stochastics.F90
+7 −7 physics/GFS_suite_interstitial.F90
+34 −22 physics/GFS_surface_composites.F90
+13 −24 physics/GFS_surface_generic.F90
+18 −90 physics/GFS_surface_generic.meta
+1 −1 physics/gcycle.F90
+8 −8 physics/sfc_cice.meta
+6 −6 physics/sfc_nst.f
+18 −0 physics/sfc_nst.meta
87 changes: 87 additions & 0 deletions ccpp/suites/suite_FV3_GFS_2017_satmedmf_coupled.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
<?xml version="1.0" encoding="UTF-8"?>

<suite name="FV3_GFS_2017_satmedmf_coupled" lib="ccppphys" ver="3">
<!-- <init></init> -->
<group name="time_vary">
<subcycle loop="1">
<scheme>GFS_time_vary_pre</scheme>
<scheme>GFS_rrtmg_setup</scheme>
<scheme>GFS_rad_time_vary</scheme>
<scheme>GFS_phys_time_vary</scheme>
</subcycle>
</group>
<group name="radiation">
<subcycle loop="1">
<scheme>GFS_suite_interstitial_rad_reset</scheme>
<scheme>GFS_rrtmg_pre</scheme>
<scheme>rrtmg_sw_pre</scheme>
<scheme>rrtmg_sw</scheme>
<scheme>rrtmg_sw_post</scheme>
<scheme>rrtmg_lw_pre</scheme>
<scheme>rrtmg_lw</scheme>
<scheme>rrtmg_lw_post</scheme>
<scheme>GFS_rrtmg_post</scheme>
</subcycle>
</group>
<group name="physics">
<subcycle loop="1">
<scheme>GFS_suite_interstitial_phys_reset</scheme>
<scheme>GFS_suite_stateout_reset</scheme>
<scheme>get_prs_fv3</scheme>
<scheme>GFS_suite_interstitial_1</scheme>
<scheme>GFS_surface_generic_pre</scheme>
<scheme>GFS_surface_composites_pre</scheme>
<scheme>dcyc2t3</scheme>
<scheme>GFS_surface_composites_inter</scheme>
<scheme>GFS_suite_interstitial_2</scheme>
</subcycle>
<!-- Surface iteration loop -->
<subcycle loop="2">
<scheme>sfc_diff</scheme>
<scheme>GFS_surface_loop_control_part1</scheme>
<scheme>sfc_ocean</scheme>
<scheme>lsm_noah</scheme>
<scheme>sfc_cice</scheme>
<scheme>sfc_sice</scheme>
<scheme>GFS_surface_loop_control_part2</scheme>
</subcycle>
<!-- End of surface iteration loop -->
<subcycle loop="1">
<scheme>GFS_surface_composites_post</scheme>
<scheme>sfc_diag</scheme>
<scheme>sfc_diag_post</scheme>
<scheme>GFS_surface_generic_post</scheme>
<scheme>GFS_PBL_generic_pre</scheme>
<scheme>satmedmfvdif</scheme>
<scheme>GFS_PBL_generic_post</scheme>
<scheme>GFS_GWD_generic_pre</scheme>
<scheme>cires_ugwp</scheme>
<scheme>cires_ugwp_post</scheme>
<scheme>GFS_GWD_generic_post</scheme>
<scheme>rayleigh_damp</scheme>
<scheme>GFS_suite_stateout_update</scheme>
<scheme>ozphys</scheme>
<scheme>GFS_DCNV_generic_pre</scheme>
<scheme>get_phi_fv3</scheme>
<scheme>GFS_suite_interstitial_3</scheme>
<scheme>samfdeepcnv</scheme>
<scheme>GFS_DCNV_generic_post</scheme>
<scheme>GFS_SCNV_generic_pre</scheme>
<scheme>samfshalcnv</scheme>
<scheme>GFS_SCNV_generic_post</scheme>
<scheme>GFS_suite_interstitial_4</scheme>
<scheme>cnvc90</scheme>
<scheme>GFS_MP_generic_pre</scheme>
<scheme>zhaocarr_gscond</scheme>
<scheme>zhaocarr_precpd</scheme>
<scheme>GFS_MP_generic_post</scheme>
<scheme>maximum_hourly_diagnostics</scheme>
</subcycle>
</group>
<group name="stochastics">
<subcycle loop="1">
<scheme>GFS_stochastics</scheme>
</subcycle>
</group>
<!-- <finalize></finalize> -->
</suite>
92 changes: 92 additions & 0 deletions ccpp/suites/suite_FV3_GFS_v15p2_coupled.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
<?xml version="1.0" encoding="UTF-8"?>

<suite name="FV3_GFS_v15p2_coupled" lib="ccppphys" ver="3">
<!-- <init></init> -->
<group name="fast_physics">
<subcycle loop="1">
<scheme>fv_sat_adj</scheme>
</subcycle>
</group>
<group name="time_vary">
<subcycle loop="1">
<scheme>GFS_time_vary_pre</scheme>
<scheme>GFS_rrtmg_setup</scheme>
<scheme>GFS_rad_time_vary</scheme>
<scheme>GFS_phys_time_vary</scheme>
</subcycle>
</group>
<group name="radiation">
<subcycle loop="1">
<scheme>GFS_suite_interstitial_rad_reset</scheme>
<scheme>GFS_rrtmg_pre</scheme>
<scheme>rrtmg_sw_pre</scheme>
<scheme>rrtmg_sw</scheme>
<scheme>rrtmg_sw_post</scheme>
<scheme>rrtmg_lw_pre</scheme>
<scheme>rrtmg_lw</scheme>
<scheme>rrtmg_lw_post</scheme>
<scheme>GFS_rrtmg_post</scheme>
</subcycle>
</group>
<group name="physics">
<subcycle loop="1">
<scheme>GFS_suite_interstitial_phys_reset</scheme>
<scheme>GFS_suite_stateout_reset</scheme>
<scheme>get_prs_fv3</scheme>
<scheme>GFS_suite_interstitial_1</scheme>
<scheme>GFS_surface_generic_pre</scheme>
<scheme>GFS_surface_composites_pre</scheme>
<scheme>dcyc2t3</scheme>
<scheme>GFS_surface_composites_inter</scheme>
<scheme>GFS_suite_interstitial_2</scheme>
</subcycle>
<!-- Surface iteration loop -->
<subcycle loop="2">
<scheme>sfc_diff</scheme>
<scheme>GFS_surface_loop_control_part1</scheme>
<scheme>sfc_ocean</scheme>
<scheme>lsm_noah</scheme>
<scheme>sfc_cice</scheme>
<scheme>sfc_sice</scheme>
<scheme>GFS_surface_loop_control_part2</scheme>
</subcycle>
<!-- End of surface iteration loop -->
<subcycle loop="1">
<scheme>GFS_surface_composites_post</scheme>
<scheme>sfc_diag</scheme>
<scheme>sfc_diag_post</scheme>
<scheme>GFS_surface_generic_post</scheme>
<scheme>GFS_PBL_generic_pre</scheme>
<scheme>hedmf</scheme>
<scheme>GFS_PBL_generic_post</scheme>
<scheme>GFS_GWD_generic_pre</scheme>
<scheme>cires_ugwp</scheme>
<scheme>cires_ugwp_post</scheme>
<scheme>GFS_GWD_generic_post</scheme>
<scheme>rayleigh_damp</scheme>
<scheme>GFS_suite_stateout_update</scheme>
<scheme>ozphys_2015</scheme>
<scheme>h2ophys</scheme>
<scheme>GFS_DCNV_generic_pre</scheme>
<scheme>get_phi_fv3</scheme>
<scheme>GFS_suite_interstitial_3</scheme>
<scheme>samfdeepcnv</scheme>
<scheme>GFS_DCNV_generic_post</scheme>
<scheme>GFS_SCNV_generic_pre</scheme>
<scheme>samfshalcnv</scheme>
<scheme>GFS_SCNV_generic_post</scheme>
<scheme>GFS_suite_interstitial_4</scheme>
<scheme>cnvc90</scheme>
<scheme>GFS_MP_generic_pre</scheme>
<scheme>gfdl_cloud_microphys</scheme>
<scheme>GFS_MP_generic_post</scheme>
<scheme>maximum_hourly_diagnostics</scheme>
</subcycle>
</group>
<group name="stochastics">
<subcycle loop="1">
<scheme>GFS_stochastics</scheme>
</subcycle>
</group>
<!-- <finalize></finalize> -->
</suite>
93 changes: 43 additions & 50 deletions gfsphysics/GFS_layer/GFS_physics_driver.F90
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -544,9 +544,6 @@ subroutine GFS_physics_driver &
doms, psautco_l, prautco_l, ocalnirbm_cpl, ocalnirdf_cpl, &
ocalvisbm_cpl, ocalvisdf_cpl, dtzm, temrain1, t2mmp, q2mp, &
psaur_l, praur_l, &
!--- coupling inputs for physics
dtsfc_cice, dqsfc_cice, dusfc_cice, dvsfc_cice, &
! dtsfc_cice, dqsfc_cice, dusfc_cice, dvsfc_cice, ulwsfc_cice, &
!--- for CS-convection
wcbmax

Expand Down Expand Up @@ -678,16 +675,15 @@ subroutine GFS_physics_driver &
real :: pshltr,QCQ,rh02
real(kind=kind_phys), allocatable, dimension(:,:) :: den

!! Initialize local variables (mainly for debugging purposes, because the
!! corresponding variables Interstitial(nt)%... are reset to zero every time);
!! these variables are only modified over parts of the entire domain (related
!! to land surface mask etc.)
!! Initialize local variables (for debugging purposes only,
!! because the corresponding variables Interstitial(nt)%...
!! are reset to zero every time).
!snowmt = 0.
!gamq = 0.
!gamt = 0.
!gflx = 0.
!hflx = 0.
!

!! Strictly speaking, this is not required. But when
!! hunting for bit-for-bit differences, doing the same as
!! in GFS_suite_stateout_reset makes life a lot easier.
Expand Down Expand Up @@ -911,10 +907,6 @@ subroutine GFS_physics_driver &
! --- set initial quantities for stochastic physics deltas
if (Model%do_sppt) then
Tbd%dtdtr = zero
do i=1,im
Tbd%drain_cpl(i) = Coupling%rain_cpl (i)
Tbd%dsnow_cpl(i) = Coupling%snow_cpl (i)
enddo
endif

! mg, sfc-perts
Expand Down Expand Up @@ -1113,14 +1105,6 @@ subroutine GFS_physics_driver &
do i=1,im
islmsk_cice(i) = nint(Coupling%slimskin_cpl(i))
flag_cice(i) = (islmsk_cice(i) == 4)

if (flag_cice(i)) then
! ulwsfc_cice(i) = Coupling%ulwsfcin_cpl(i)
dusfc_cice(i) = Coupling%dusfcin_cpl(i)
dvsfc_cice(i) = Coupling%dvsfcin_cpl(i)
dtsfc_cice(i) = Coupling%dtsfcin_cpl(i)
dqsfc_cice(i) = Coupling%dqsfcin_cpl(i)
endif
enddo
endif
!*## CCPP ##
Expand All @@ -1145,9 +1129,8 @@ subroutine GFS_physics_driver &
endif
endif
if (fice(i) < one) then
wet(i)=.true. !some open ocean/lake water exists
wet(i)=.true. ! some open ocean/lake water exists
if (.not. Model%cplflx) Sfcprop%tsfco(i) = max(Sfcprop%tsfco(i), Sfcprop%tisfc(i), tgice)

end if
else
fice(i) = zero
Expand Down Expand Up @@ -1659,13 +1642,14 @@ subroutine GFS_physics_driver &
sbsno(i) = zero
snowc(i) = zero
snohf(i) = zero
!## CCPP ##* GFS_surface_generic.F90/GFS_surface_generic_pre_run
Diag%zlvl(i) = Statein%phil(i,1) * onebg
Diag%smcwlt2(i) = zero
Diag%smcref2(i) = zero

wind(i) = max(sqrt(Statein%ugrs(i,1)*Statein%ugrs(i,1) + &
Statein%vgrs(i,1)*Statein%vgrs(i,1)) &
+ max(zero, min(Tbd%phy_f2d(i,Model%num_p2d), 30.0)), one)
!*## CCPP ##
enddo
!*## CCPP ##

Expand Down Expand Up @@ -1909,6 +1893,30 @@ subroutine GFS_physics_driver &

endif !lsm

!! Strictly speaking, this is not required. But when
!! hunting for bit-for-bit differences, updating the
!! subsurface variables in the Sfcprop DDT makes
!! life a lot easier
!if (Model%frac_grid) then
! do k=1,lsoil
! do i=1,im
! if (dry(i)) then
! Sfcprop%smc(i,k) = smsoil(i,k)
! Sfcprop%stc(i,k) = stsoil(i,k)
! Sfcprop%slc(i,k) = slsoil(i,k)
! endif
! enddo
! enddo
!else
! do k=1,lsoil
! do i=1,im
! Sfcprop%smc(i,k) = smsoil(i,k)
! Sfcprop%stc(i,k) = stsoil(i,k)
! Sfcprop%slc(i,k) = slsoil(i,k)
! enddo
! enddo
!endif

! if (lprnt) write(0,*)' tseabeficemodel =',Sfcprop%tsfc(ipr),' me=',me &
! &, ' kdt=',kdt,' tsfc32=',tsfc3(ipr,2),' fice=',fice(ipr) &
! &,' stsoil=',stsoil(ipr,:)
Expand All @@ -1931,8 +1939,9 @@ subroutine GFS_physics_driver &
(im, Statein%tgrs(:,1), &
Statein%qgrs(:,1,1), cd3(:,2), cdq3(:,2), &
Statein%prsl(:,1), wind, &
flag_cice, flag_iter, dqsfc_cice, dtsfc_cice, &
dusfc_cice, dvsfc_cice, &
flag_cice, flag_iter, &
Coupling%dqsfcin_cpl, Coupling%dtsfcin_cpl, &
Coupling%dusfcin_cpl, Coupling%dvsfcin_cpl, &
! --- outputs:
qss3(:,2), cmm3(:,2), chh3(:,2), evap3(:,2), hflx3(:,2), &
stress3(:,2))
Expand Down Expand Up @@ -2103,15 +2112,11 @@ subroutine GFS_physics_driver &
ep1d(i) = ep1d3(i,k)
Sfcprop%weasd(i) = weasd3(i,k)
Sfcprop%snowd(i) = snowd3(i,k)

evap(i) = evap3(i,k)
hflx(i) = hflx3(i,k)
qss(i) = qss3(i,k)
Sfcprop%tsfc(i) = tsfc3(i,k)

Diag%cmm(i) = cmm3(i,k)
Diag%chh(i) = chh3(i,k)

Sfcprop%zorll(i) = zorl3(i,1)
Sfcprop%zorlo(i) = zorl3(i,3)

Expand All @@ -2120,7 +2125,6 @@ subroutine GFS_physics_driver &
txo = one - txi
evap(i) = txi * evap3(i,2) + txo * evap3(i,3)
hflx(i) = txi * hflx3(i,2) + txo * hflx3(i,3)
! Sfcprop%tsfc(i) = txi * tice(i) + txo * tsfc3(i,3)
Sfcprop%tsfc(i) = txi * tsfc3(i,2) + txo * tsfc3(i,3)
else ! return updated lake ice thickness & concentration to global array
if (islmsk(i) == 2) then
Expand Down Expand Up @@ -2839,10 +2843,10 @@ subroutine GFS_physics_driver &
do i=1,im
if (Sfcprop%oceanfrac(i) > zero) then ! Ocean only, NO LAKES
if (Sfcprop%fice(i) > one - epsln) then ! no open water, thus use results from CICE
Coupling%dusfci_cpl(i) = dusfc_cice(i)
Coupling%dvsfci_cpl(i) = dvsfc_cice(i)
Coupling%dtsfci_cpl(i) = dtsfc_cice(i)
Coupling%dqsfci_cpl(i) = dqsfc_cice(i)
Coupling%dusfci_cpl(i) = Coupling%dusfcin_cpl(i)
Coupling%dvsfci_cpl(i) = Coupling%dvsfcin_cpl(i)
Coupling%dtsfci_cpl(i) = Coupling%dtsfcin_cpl(i)
Coupling%dqsfci_cpl(i) = Coupling%dqsfcin_cpl(i)
elseif (icy(i) .or. dry(i)) then ! use stress_ocean from sfc_diff for opw component at mixed point
tem1 = max(Diag%q1(i), 1.e-8)
rho = Statein%prsl(i,1) / (con_rd*Diag%t1(i)*(one+con_fvirt*tem1))
Expand All @@ -2856,8 +2860,6 @@ subroutine GFS_physics_driver &
endif
Coupling%dtsfci_cpl(i) = con_cp * rho * hflx3(i,3) ! sensible heat flux over open ocean
Coupling%dqsfci_cpl(i) = con_hvap * rho * evap3(i,3) ! latent heat flux over open ocean
! if (lprnt .and. i == ipr) write(0,*)' hflx33=',hflx3(i,3),' evap33=',evap3(i,3), &
! ' con_cp=',con_cp,' rho=',rho,' con_hvap=',con_hvap
else ! use results from PBL scheme for 100% open ocean
Coupling%dusfci_cpl(i) = dusfc1(i)
Coupling%dvsfci_cpl(i) = dvsfc1(i)
Expand Down Expand Up @@ -5283,7 +5285,7 @@ subroutine GFS_physics_driver &
!*## CCPP ##
!## CCPP ##* GFS_MP_generic.F90/GFS_MP_generic_post_run
Diag%rain(:) = Diag%rainc(:) + frain * rain1(:) ! total rain per timestep

! --- get the amount of different precip type for Noah MP
! --- convert from m/dtp to mm/s
if (Model%lsm==Model%lsm_noahmp) then
Expand Down Expand Up @@ -5463,10 +5465,10 @@ subroutine GFS_physics_driver &

if (Model%cplflx .or. Model%cplchm) then
do i = 1, im
Coupling%rain_cpl(i) = Coupling%rain_cpl(i) &
+ Diag%rain(i) * (one-Sfcprop%srflag(i))
Coupling%snow_cpl(i) = Coupling%snow_cpl(i) &
+ Diag%rain(i) * Sfcprop%srflag(i)
Tbd%drain_cpl(i)= Diag%rain(i) * (one-Sfcprop%srflag(i))
Comment thread
junwang-noaa marked this conversation as resolved.
Tbd%dsnow_cpl(i)= Diag%rain(i) * Sfcprop%srflag(i)
Coupling%rain_cpl(i) = Coupling%rain_cpl(i) + Tbd%drain_cpl(i)
Coupling%snow_cpl(i) = Coupling%snow_cpl(i) + Tbd%dsnow_cpl(i)
enddo
endif

Expand Down Expand Up @@ -5557,15 +5559,6 @@ subroutine GFS_physics_driver &
if (Model%do_sppt) then
!--- radiation heating rate
Tbd%dtdtr(1:im,:) = Tbd%dtdtr(1:im,:) + dtdtc(1:im,:)*dtf
do i = 1, im
if (t850(i) > 273.16) then
!--- change in change in rain precip
Tbd%drain_cpl(i) = Diag%rain(i) - Tbd%drain_cpl(i)
else
!--- change in change in snow precip
Tbd%dsnow_cpl(i) = Diag%rain(i) - Tbd%dsnow_cpl(i)
endif
enddo
endif
!*## CCPP ##
!## CCPP ##* This block is not in the CCPP since it is not needed in the CCPP.
Expand Down
Loading