Update ccpp-physics. Make RRTMGP thread safe#418
Conversation
…r-model into hotfix_GPthreading
junwang-noaa
left a comment
junwang-noaa
left a comment
There was a problem hiding this comment.
We still need orion log file, the code can be committed with that.
@junwang-noaa CI tests passed. I'll quickly hop on orion and run the tests. |
* upstream/develop: update MOM6 to GFDL 20210224 main branch commit (ufs-community#439) Add GNU and Cheyenne Support to Automated RT (ufs-community#444) Move Noah MP init to CCPP and update Noah MP regression tests, ice flux init bug fix in CCPP (ufs-community#425) Feature/rt automation (ufs-community#403) Update ccpp-physics. Make RRTMGP thread safe (ufs-community#418) Update regression tests from GFSv15+Thompson to GFSv16+Thompson, include "Add one regional regression test in DEBUG mode. (ufs-community#419)" (ufs-community#421) UGWP v0 v1 combined (ufs-community#396) add optional mesh in MOM6; add dz_min and min_seaice as configurable variables for coupled model (ufs-community#399) updates FMS to 2020.04.01 (ufs-community#392) Move LSM vegetation lookup tables into CCPP, clean up RUC snow cover on ice initialization (remove IPD step 2) (ufs-community#407) Update CMEPS for HAFS integration; add datm and coupled-model tests on Gaea (ufs-community#401) Remove legacy gnumake build from fv3atm and NEMS, remove legacy Python 2.7 support, rename v16beta to v16 and RT updates (ufs-community#384) MOM6 bugfixes, GFDL update, update CDMBGWD settings; fix for restart reproducibility (without waves) when USE_LA_LI2016=True, sign error on fprec passed to ocean, GFDL update, resolution dependent cdmbgwd settings (ufs-community#379) dycore options to add zero-gradient BC to reconstruct interface u/v and change dz_min as input (ufs-community#369) Update develop from NOAA-GSL: RUC ice, MYNN sfclay, stochastic land perturbations (ufs-community#386) update cpl gfsv16 tests, rrtmgp fix and bug fixes in cmeps (ufs-community#378) point fv3 to EMC develop branch (ufs-community#377) Remove IPD steps 3 and 5 (ufs-community#357) Update CMEPS (ufs-community#345) Implementation of CCPP timestep_init and timestep_final phases (ufs-community#337) Remove unnecessary SIMD instruction sets for Jet, first round of cleanup in rt.conf, initialize cld_amt to zero for regional runs (dycore) (ufs-community#353) add frac grid input, update and add additional cpld tests (ufs-community#354) Add checkpoint restarts for ufs-cpld (ufs-community#342) Update the format of rt.conf (ufs-community#349) Remove IPD (step 1) (ufs-community#331) Feature/ww3update (ufs-community#334) Replace old regional SDF with FV3_GFS_v15_thompson_mynn (ufs-community#333) Update modules with hpc-stack v1.1.0 (ufs-community#319) Regression test log for PR ufs-community#323 for jet.intel (ufs-community#336) RRTMGP and Thompson MP coupling (ufs-community#323) Add 2 new tests for DATM-MOM6-CICE6 application (ufs-community#332) Add optional bulk flux calculation in ufs-datm (ufs-community#266) Final-final GFS v16 updates / restart reproducibility bugfixes (ufs-community#325) Updates to build for JEDI linking/control, add wcoss2 (ufs-community#295) Update CICE, Move regression test input outside baseline directory (ufs-community#270) Feature/update mom6 and retain b4b results for 025x025 resolution (ufs-community#290) Update for Jet, bug fixes in running with frac_grid=T and GFDL MP, and in restarting with frac_grid=T (ufs-community#304) Updates to stochastic_physics_wrapper (ufs-community#280) Update develop from gsd/develop 2020/11/20: Unified gravity wave drag, updates to other GSL physics (ufs-community#297) Fix to allow quilting with non-factors for layout (ufs-community#250) rt update (ufs-community#261)
|
I know this conversation is from some time ago but I'm wondering if the code modifications noted here are in the development branch. We are running UFS on WCOSS2 HPE/Cray systems to evaluate RRTMGP radiation scheme and find that it fails in mo_gas_concentrat routine in longwave code if more than 1 OMP thread is used. shortwave code (and other RRTMGP routines executed before that) are OK. I see, in the initial comment in this conversation, mention of adding OMP CRITICAL directives for thread safety (among other changes), however I do not see any OMP CRITICAL directives in the RRTMGP radiation code in the latest development branch. Comments welcome. Thanks, |
|
@pj-gdit It's been awhile, but I recall that we found a solution that didn't require adding OMP directives to the RRTMGp scheme. May I ask that you open a brief issue in the UFS physics repository? |
5 participants
This PR contains several changes to the initialization and setup of the RRTMGP radiation scheme to allow for use across multiple openMP threads.
*) Moved Interstitial RRTMGP DDTs (ty_rrtmgp_gas_optics, ty_cloud_optics) to static fields defined during initialization in module memory.
*) Add "$omp critical" statements around the calling type-bound procedures during initialization.
*) Move allocation of ty_gas_conc from Interstitial to GFS_typedefs. Initialize ty_gas_concs for all blocks during initialization.
This issue was brought to our attention by @yangfanglin and Qingfu Liu at EMC while testing RRTMGP in high-resolution cases, which require multiple threads.
The GP regression tests on hera using Intel were successful.
There are no changes to the baselines
Waiting on
NCAR/ccpp-physics#568
NOAA-EMC/ufsatm#247