Skip to content

RRTMGP in ufs-weather-model #29

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
climbfuji merged 9 commits into from
Mar 26, 2020
Merged

RRTMGP in ufs-weather-model #29

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
44604a0
Update EMC develop from NCAR dtc/develop 2020/03/17 (#84)
climbfuji Mar 19, 2020
acd17fc
Update .gitmodules and submodule pointer for fv3atm for code review a…
climbfuji Mar 18, 2020
b6d9813
Add RRTMGP regression tests in DEBUG and PROD mode, update compile_cm…
climbfuji Mar 19, 2020
bf0a2c4
Merge branch 'develop' of https://github.com/ufs-community/ufs-weathe…
climbfuji Mar 19, 2020
e04023e
Update submodule pointer for FV3 after RRTMGP bugfixes
climbfuji Mar 23, 2020
a799e29
Update submodule pointer for fv3atm after merging Python 3 compatibil…
climbfuji Mar 25, 2020
9a647bd
Add REPRO test for fv3_ccpp_rrtmgp
climbfuji Mar 25, 2020
55936be
Update submodule pointer for FV3 following code review changes
climbfuji Mar 26, 2020
6d42d95
Revert change .gitmodules and update submodule pointer for fv3atm
climbfuji Mar 26, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions .gitmodules
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,8 @@
branch = dtc/develop
[submodule "NEMS"]
path = NEMS
#url = https://github.com/NOAA-EMC/NEMS
#branch = develop
url = https://github.com/climbfuji/NEMS
branch = final_pr_before_merging_to_develop_or_master_20200313
url = https://github.com/NOAA-EMC/NEMS
branch = develop
[submodule "FMS"]
path = FMS
url = https://github.com/NOAA-GFDL/FMS
Expand Down
2 changes: 1 addition & 1 deletion FV3
Open in desktop
Submodule FV3 updated 13 files
+1 −0 CMakeLists.txt
+6 −6 ccpp/build_ccpp.sh
+112 −2 ccpp/config/ccpp_prebuild_config.py
+1 −1 ccpp/framework
+1 −1 ccpp/physics
+95 −0 ccpp/suites/suite_FV3_GFS_2017_RRTMGP.xml
+0 −6 gfsphysics/CMakeLists.txt
+274 −18 gfsphysics/GFS_layer/GFS_typedefs.F90
+641 −0 gfsphysics/GFS_layer/GFS_typedefs.meta
+4 −9 gfsphysics/makefile
+4 −0 io/CMakeLists.txt
+4 −0 ipd/CMakeLists.txt
+4 −0 stochastic_physics/CMakeLists.txt
2 changes: 1 addition & 1 deletion NEMS
Open in desktop
Submodule NEMS updated 0 files
1 change: 0 additions & 1 deletion compsets/all.input
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ run fv3_gfdlmprad @ fv3, standard, baseline, ww3, plat==hera.intel
run fv3_gfdlmprad_atmwav @ fv3, standard, baseline, ww3, plat==hera.intel,plat==wcoss_dell_p3,plat==wcoss.cray
run fv3_gfdlmprad_gwd @ fv3, standard, baseline
run fv3_gfdlmprad_noahmp @ fv3, standard, baseline
run fv3_thompson @ fv3, standard, baseline
run fv3_wsm6 @ fv3, standard, baseline
run fv3_csawmgshoc @ fv3, standard, baseline
run fv3_rasmgshoc @ fv3, standard, baseline
Expand Down
984 changes: 492 additions & 492 deletions log/report-hera.intel-log/build_fv3.exe.log
View file
Open in desktop

Large diffs are not rendered by default.

1,004 changes: 502 additions & 502 deletions log/report-hera.intel-log/build_fv3_32bit.exe.log
View file
Open in desktop

Large diffs are not rendered by default.

982 changes: 491 additions & 491 deletions log/report-hera.intel-log/build_fv3_32bit_debug.exe.log
View file
Open in desktop

Large diffs are not rendered by default.

878 changes: 439 additions & 439 deletions log/report-hera.intel-log/build_fv3_appbuilder.exe.log
View file
Open in desktop

Large diffs are not rendered by default.

1,145 changes: 567 additions & 578 deletions log/report-hera.intel-log/build_fv3_ww3.exe.log
View file
Open in desktop

Large diffs are not rendered by default.

350 changes: 174 additions & 176 deletions log/report-hera.intel-log/rtreport.txt
View file
Open in desktop

Large diffs are not rendered by default.

1,032 changes: 516 additions & 516 deletions log/report-wcoss.cray-log/build_fv3.exe.log
View file
Open in desktop

Large diffs are not rendered by default.

1,064 changes: 532 additions & 532 deletions log/report-wcoss.cray-log/build_fv3_32bit.exe.log
View file
Open in desktop

Large diffs are not rendered by default.

932 changes: 466 additions & 466 deletions log/report-wcoss.cray-log/build_fv3_appbuilder.exe.log
View file
Open in desktop

Large diffs are not rendered by default.

1,244 changes: 624 additions & 620 deletions log/report-wcoss.cray-log/build_fv3_ww3.exe.log
View file
Open in desktop

Large diffs are not rendered by default.

320 changes: 160 additions & 160 deletions log/report-wcoss.cray-log/rtreport.txt
View file
Open in desktop

Large diffs are not rendered by default.

980 changes: 490 additions & 490 deletions log/report-wcoss_dell_p3-log/build_fv3.exe.log
View file
Open in desktop

Large diffs are not rendered by default.

996 changes: 498 additions & 498 deletions log/report-wcoss_dell_p3-log/build_fv3_32bit.exe.log
View file
Open in desktop

Large diffs are not rendered by default.

886 changes: 443 additions & 443 deletions log/report-wcoss_dell_p3-log/build_fv3_appbuilder.exe.log
View file
Open in desktop

Large diffs are not rendered by default.

1,126 changes: 559 additions & 567 deletions log/report-wcoss_dell_p3-log/build_fv3_ww3.exe.log
View file
Open in desktop

Large diffs are not rendered by default.

433 changes: 180 additions & 253 deletions log/report-wcoss_dell_p3-log/rtreport.txt
View file
Open in desktop

Large diffs are not rendered by default.

10 changes: 10 additions & 0 deletions parm/ccpp_control.nml.IN
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,16 @@
nspinup = 1000
iseed_ca = 0
isppt_deep = .false.
do_RRTMGP = @[DO_RRTMGP]
Copy link
Copy Markdown
Collaborator

@grantfirl grantfirl Mar 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just asking, but is ccpp_control the best namelist to put this in? Should there be a separate one for RRTMGP? I see that DO_RRTMGP is set externally such that the other RRTMGP variables become moot if it is set to false.

Copy link
Copy Markdown
Author

@climbfuji climbfuji Mar 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want to avoid a proliferation of namelist templates in parm/ - something that EMC is adamant about for good reasons. Therefore, this makes a lot of sense. Is also done for NoahMP.

Copy link
Copy Markdown
Collaborator

@grantfirl grantfirl Mar 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. This seems to mark a change from past behavior.

Copy link
Copy Markdown
Author

@climbfuji climbfuji Mar 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, not. We were asked to combine those as much as it makes sense more than a year ago when merging the initial CCPP version in July 2019.

active_gases = 'h2o_co2_o3_n2o_ch4_o2'
ngases = 6
lw_file_gas = 'rrtmgp-data-lw-g256-2018-12-04.nc'
lw_file_clouds = 'rrtmgp-cloud-optics-coeffs-lw.nc'
sw_file_gas = 'rrtmgp-data-sw-g224-2018-12-04.nc'
sw_file_clouds = 'rrtmgp-cloud-optics-coeffs-sw.nc'
rrtmgp_cld_optics = 0
rrtmgp_nrghice = 3
rrtmgp_ngauss_ang = 3
/

&interpolator_nml
Expand Down
73,312 changes: 66,384 additions & 6,928 deletions tests/Compile_hera.intel.log
View file
Open in desktop

Large diffs are not rendered by default.

11,232 changes: 4,541 additions & 6,691 deletions tests/Compile_wcoss_cray.log
View file
Open in desktop

Large diffs are not rendered by default.

56,250 changes: 54,971 additions & 1,279 deletions tests/Compile_wcoss_dell_p3.log
View file
Open in desktop

Large diffs are not rendered by default.

6,024 changes: 5,825 additions & 199 deletions tests/RegressionTests_hera.intel.log
View file
Open in desktop

Large diffs are not rendered by default.

134 changes: 67 additions & 67 deletions tests/RegressionTests_wcoss_cray.log
View file
Open in desktop

Large diffs are not rendered by default.

5,309 changes: 5,122 additions & 187 deletions tests/RegressionTests_wcoss_dell_p3.log
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions tests/compile_cmake.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@ if [[ "${MAKE_OPT}" == *"CCPP=Y"* ]]; then
)

# Read list of schemes, caps, and static API
source ${BUILD_DIR}/FV3/ccpp/physics/CCPP_TYPEDEFS.sh
source ${BUILD_DIR}/FV3/ccpp/physics/CCPP_SCHEMES.sh
source ${BUILD_DIR}/FV3/ccpp/physics/CCPP_CAPS.sh
source ${BUILD_DIR}/FV3/ccpp/physics/CCPP_STATIC_API.sh
Expand Down
3 changes: 3 additions & 0 deletions tests/default_vars.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,9 @@ export MAKE_NH=.T.
export MOUNTAIN=.F.
export NA_INIT=1

# Radiation
export DO_RRTMGP=.F.

# Microphysics
export IMP_PHYSICS=11
# GFDL MP
Expand Down
4 changes: 4 additions & 0 deletions tests/fv3_conf/ccpp_control_run.IN
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,10 @@ cp @[RTPWD]/WW3_input_data/@[SYEAR]@[SMONTH]@[SDAY]/ww3_multi.inp .
cp @[RTPWD]/WW3_input_data/@[SYEAR]@[SMONTH]@[SDAY]/rmp_src* .
fi

if [ $DO_RRTMGP = .T. ]; then
cp @[RTPWD]/RRTMGP_input_data/* .
fi

cp ${PATHRT}/../FV3/ccpp/suites/suite_${CCPP_SUITE}.xml suite_${CCPP_SUITE}.xml

# Add path to libccpp.so and libccpphys.so to LD_LIBRARY_PATH, append to module-setup.sh
Expand Down
10 changes: 6 additions & 4 deletions tests/rt.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ RUN | fv3_read_inc
RUN | fv3_gfdlmp | standard | | fv3 |
RUN | fv3_gfdlmprad_gwd | standard | | fv3 |
RUN | fv3_gfdlmprad_noahmp | standard | | fv3 |
#RUN | fv3_thompson | standard | | fv3 |
RUN | fv3_wsm6 | standard | | fv3 |
RUN | fv3_wrtGauss_netcdf_esmf | standard | | fv3 |
RUN | fv3_wrtGauss_netcdf | standard | | fv3 |
Expand Down Expand Up @@ -374,19 +373,21 @@ RUN | fv3_ccpp_thompson_no_aero
# CCPP PROD tests #
#######################################################################################################################################################################################

COMPILE | CCPP=Y STATIC=Y SUITES=FV3_GFS_v15p2,FV3_GFS_v16beta | standard | hera.intel | fv3 |
COMPILE | CCPP=Y STATIC=Y SUITES=FV3_GFS_v15p2,FV3_GFS_v16beta,FV3_GFS_2017_RRTMGP | standard | hera.intel | fv3 |

RUN | fv3_ccpp_gfs_v15p2 | standard | hera.intel | fv3 |
RUN | fv3_ccpp_gfs_v16beta | standard | hera.intel | fv3 |
RUN | fv3_ccpp_rrtmgp | standard | hera.intel | fv3 |

#######################################################################################################################################################################################
# CCPP DEBUG tests #
#######################################################################################################################################################################################

COMPILE | CCPP=Y STATIC=Y SUITES=FV3_GFS_v15p2,FV3_GFS_v16beta DEBUG=Y | standard | hera.intel | fv3 |
COMPILE | CCPP=Y STATIC=Y SUITES=FV3_GFS_v15p2,FV3_GFS_v16beta,FV3_GFS_2017_RRTMGP DEBUG=Y | standard | hera.intel | fv3 |

RUN | fv3_ccpp_gfs_v15p2_debug | standard | hera.intel | fv3 |
RUN | fv3_ccpp_gfs_v16beta_debug | standard | hera.intel | fv3 |
RUN | fv3_ccpp_rrtmgp_debug | standard | hera.intel | fv3 |

COMPILE | CCPP=Y STATIC=Y SUITES=FV3_GSD_v0,FV3_GFS_v15_thompson 32BIT=Y DEBUG=Y | standard | wcoss_cray | fv3 |
COMPILE | CCPP=Y STATIC=Y SUITES=FV3_GSD_v0,FV3_GFS_v15_thompson 32BIT=Y DEBUG=Y | standard | wcoss_dell_p3 | fv3 |
Expand All @@ -402,10 +403,11 @@ RUN | fv3_ccpp_thompson_no_aero_debug
# CCPP REPRO tests #
#######################################################################################################################################################################################

COMPILE | CCPP=Y STATIC=Y SUITES=FV3_GFS_v15p2,FV3_GFS_v16beta REPRO=Y | standard | hera.intel | fv3 |
COMPILE | CCPP=Y STATIC=Y SUITES=FV3_GFS_v15p2,FV3_GFS_v16beta,FV3_GFS_2017_RRTMGP REPRO=Y | standard | hera.intel | fv3 |

RUN | fv3_ccpp_gfs_v15p2 | standard | hera.intel | fv3 |
RUN | fv3_ccpp_gfs_v16beta | standard | hera.intel | fv3 |
RUN | fv3_ccpp_rrtmgp | standard | hera.intel | fv3 |

#######################################################################################################################################################################################
# IPD REPRO tests (verify against CCPP REPRO tests) #
Expand Down
6 changes: 3 additions & 3 deletions tests/rt.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -383,11 +383,11 @@ done

# Fix me - make those definitions and DISKNM consistent
if [[ $MACHINE_ID = hera.* ]]; then
RTPWD=${RTPWD:-$DISKNM/NEMSfv3gfs/develop-20200317/${COMPILER^^}}
RTPWD=${RTPWD:-$DISKNM/NEMSfv3gfs/develop-20200319/${COMPILER^^}}
elif [[ $MACHINE_ID = cheyenne.* ]]; then
RTPWD=${RTPWD:-$DISKNM/develop-20200317/${COMPILER^^}}
RTPWD=${RTPWD:-$DISKNM/develop-20200319/${COMPILER^^}}
else
RTPWD=${RTPWD:-$DISKNM/NEMSfv3gfs/develop-20200317}
RTPWD=${RTPWD:-$DISKNM/NEMSfv3gfs/develop-20200319}
fi

shift $((OPTIND-1))
Expand Down
84 changes: 84 additions & 0 deletions tests/tests/fv3_ccpp_rrtmgp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
###############################################################################
#
# FV3 CCPP RRTMGP test
#
###############################################################################

export TEST_DESCR="Compare FV3 CCPP RRTMGP results with previous trunk version"

export CNTL_DIR=fv3_rrtmgp

export LIST_FILES="atmos_4xdaily.tile1.nc \
atmos_4xdaily.tile2.nc \
atmos_4xdaily.tile3.nc \
atmos_4xdaily.tile4.nc \
atmos_4xdaily.tile5.nc \
atmos_4xdaily.tile6.nc \
phyf000.tile1.nc \
phyf000.tile2.nc \
phyf000.tile3.nc \
phyf000.tile4.nc \
phyf000.tile5.nc \
phyf000.tile6.nc \
phyf024.tile1.nc \
phyf024.tile2.nc \
phyf024.tile3.nc \
phyf024.tile4.nc \
phyf024.tile5.nc \
phyf024.tile6.nc \
dynf000.tile1.nc \
dynf000.tile2.nc \
dynf000.tile3.nc \
dynf000.tile4.nc \
dynf000.tile5.nc \
dynf000.tile6.nc \
dynf024.tile1.nc \
dynf024.tile2.nc \
dynf024.tile3.nc \
dynf024.tile4.nc \
dynf024.tile5.nc \
dynf024.tile6.nc \
RESTART/coupler.res \
RESTART/fv_core.res.nc \
RESTART/fv_core.res.tile1.nc \
RESTART/fv_core.res.tile2.nc \
RESTART/fv_core.res.tile3.nc \
RESTART/fv_core.res.tile4.nc \
RESTART/fv_core.res.tile5.nc \
RESTART/fv_core.res.tile6.nc \
RESTART/fv_srf_wnd.res.tile1.nc \
RESTART/fv_srf_wnd.res.tile2.nc \
RESTART/fv_srf_wnd.res.tile3.nc \
RESTART/fv_srf_wnd.res.tile4.nc \
RESTART/fv_srf_wnd.res.tile5.nc \
RESTART/fv_srf_wnd.res.tile6.nc \
RESTART/fv_tracer.res.tile1.nc \
RESTART/fv_tracer.res.tile2.nc \
RESTART/fv_tracer.res.tile3.nc \
RESTART/fv_tracer.res.tile4.nc \
RESTART/fv_tracer.res.tile5.nc \
RESTART/fv_tracer.res.tile6.nc \
RESTART/phy_data.tile1.nc \
RESTART/phy_data.tile2.nc \
RESTART/phy_data.tile3.nc \
RESTART/phy_data.tile4.nc \
RESTART/phy_data.tile5.nc \
RESTART/phy_data.tile6.nc \
RESTART/sfc_data.tile1.nc \
RESTART/sfc_data.tile2.nc \
RESTART/sfc_data.tile3.nc \
RESTART/sfc_data.tile4.nc \
RESTART/sfc_data.tile5.nc \
RESTART/sfc_data.tile6.nc"

export_fv3
export NODES=$(expr $TASKS / $TPN + 1)

export DO_RRTMGP=.T.

export FV3_RUN=ccpp_control_run.IN
export CCPP_SUITE=FV3_GFS_2017_RRTMGP
export CCPP_LIB_DIR=ccpp/lib
export INPUT_NML=ccpp_control.nml.IN

RUN_SCRIPT=rt_fv3.sh
85 changes: 85 additions & 0 deletions tests/tests/fv3_ccpp_rrtmgp_debug
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
###############################################################################
#
# FV3 CCPP RRTMGP test in DEBUG mode
#
###############################################################################

export TEST_DESCR="Compare FV3 CCPP RRTMGP results in DEBUG mode with previous trunk version"

export CNTL_DIR=fv3_rrtmgp_debug

export LIST_FILES="atmos_4xdaily.tile1.nc \
atmos_4xdaily.tile2.nc \
atmos_4xdaily.tile3.nc \
atmos_4xdaily.tile4.nc \
atmos_4xdaily.tile5.nc \
atmos_4xdaily.tile6.nc \
phyf000.tile1.nc \
phyf000.tile2.nc \
phyf000.tile3.nc \
phyf000.tile4.nc \
phyf000.tile5.nc \
phyf000.tile6.nc \
phyf006.tile1.nc \
phyf006.tile2.nc \
phyf006.tile3.nc \
phyf006.tile4.nc \
phyf006.tile5.nc \
phyf006.tile6.nc \
dynf000.tile1.nc \
dynf000.tile2.nc \
dynf000.tile3.nc \
dynf000.tile4.nc \
dynf000.tile5.nc \
dynf000.tile6.nc \
dynf006.tile1.nc \
dynf006.tile2.nc \
dynf006.tile3.nc \
dynf006.tile4.nc \
dynf006.tile5.nc \
dynf006.tile6.nc \
RESTART/coupler.res \
RESTART/fv_core.res.nc \
RESTART/fv_core.res.tile1.nc \
RESTART/fv_core.res.tile2.nc \
RESTART/fv_core.res.tile3.nc \
RESTART/fv_core.res.tile4.nc \
RESTART/fv_core.res.tile5.nc \
RESTART/fv_core.res.tile6.nc \
RESTART/fv_srf_wnd.res.tile1.nc \
RESTART/fv_srf_wnd.res.tile2.nc \
RESTART/fv_srf_wnd.res.tile3.nc \
RESTART/fv_srf_wnd.res.tile4.nc \
RESTART/fv_srf_wnd.res.tile5.nc \
RESTART/fv_srf_wnd.res.tile6.nc \
RESTART/fv_tracer.res.tile1.nc \
RESTART/fv_tracer.res.tile2.nc \
RESTART/fv_tracer.res.tile3.nc \
RESTART/fv_tracer.res.tile4.nc \
RESTART/fv_tracer.res.tile5.nc \
RESTART/fv_tracer.res.tile6.nc \
RESTART/phy_data.tile1.nc \
RESTART/phy_data.tile2.nc \
RESTART/phy_data.tile3.nc \
RESTART/phy_data.tile4.nc \
RESTART/phy_data.tile5.nc \
RESTART/phy_data.tile6.nc \
RESTART/sfc_data.tile1.nc \
RESTART/sfc_data.tile2.nc \
RESTART/sfc_data.tile3.nc \
RESTART/sfc_data.tile4.nc \
RESTART/sfc_data.tile5.nc \
RESTART/sfc_data.tile6.nc"

export_fv3
export NODES=$(expr $TASKS / $TPN + 1)

export FHMAX="06"
export DO_RRTMGP=.T.

export FV3_RUN=ccpp_control_run.IN
export CCPP_SUITE=FV3_GFS_2017_RRTMGP
export CCPP_LIB_DIR=ccpp/lib
export INPUT_NML=ccpp_control.nml.IN

RUN_SCRIPT=rt_fv3.sh