Skip to content

Main candidate from NCAR (04/21/2021) #1381

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
marshallward merged 45 commits into from
May 3, 2021
Merged

Main candidate from NCAR (04/21/2021) #1381

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
0361c2e
Merge pull request #169 from gustavo-marques/delete_fotgotten_code_LBD
gustavo-marques Dec 16, 2020
170b4be
Fixes inconsistency in the calculation of tendency
gustavo-marques Dec 23, 2020
99dcfbb
Calculate tendency using fluxes
gustavo-marques Dec 28, 2020
49c0167
Merge pull request #170 from gustavo-marques/fix_lbd_tendency
alperaltuntas Dec 29, 2020
280e98b
Added a counter-based PRNG to MOM_random
iangrooms Jan 4, 2021
62138be
Used int64 from F2003 and added doc strings
iangrooms Jan 5, 2021
dde80ba
Avoids calling boundary_k_range at land pts
gustavo-marques Jan 7, 2021
20a35db
Merge branch 'pr172' into dev/ncar
alperaltuntas Jan 7, 2021
28d791e
Merge pull request #172 from gustavo-marques/fix_bug_neutral_diff
alperaltuntas Jan 8, 2021
1c8a810
Merge pull request #171 from iangrooms/dev/ncar
alperaltuntas Jan 8, 2021
0262e93
remove omp parallel do directive for a do loop including cpu_clock calls
alperaltuntas Jan 8, 2021
cc2b999
Merge pull request #173 from NCAR/fix_omp_set_diff
gustavo-marques Jan 9, 2021
c3ea9d0
Fixes latent heat from fprec and frunoff
gustavo-marques Jan 21, 2021
5837db6
Merge branch 'pr174' into dev/ncar
alperaltuntas Jan 21, 2021
230b839
Merge pull request #174 from gustavo-marques/latent_heat_fix
alperaltuntas Jan 21, 2021
e209c0e
incorporated flux correction factors
Jan 27, 2021
ed58876
Merge branch 'dev/ncar' of https://github.com/NCAR/MOM6 into dev/ncar
alperaltuntas Feb 22, 2021
b85db47
resolve minor conflicts while merging main candidate 2021-02-19 into …
alperaltuntas Feb 22, 2021
d29db6c
Merge branch 'pr1328_2' into main_candidate_2021-02-19
alperaltuntas Feb 24, 2021
387166b
reduce the line length in mom_surface_forcing_mct
alperaltuntas Feb 24, 2021
1e2fbf4
Add missing areacor
gustavo-marques Feb 25, 2021
87298d7
Fix LBD module after merging main_candidate_2021-02-19
gustavo-marques Feb 25, 2021
0252b7c
Merge branch 'main_candidate_2021-02-19' into fix_lbd_main_candidate_…
gustavo-marques Feb 25, 2021
a6cdca9
Merge pull request #178 from gustavo-marques/fix_lbd_main_candidate_2…
alperaltuntas Feb 25, 2021
9ced10b
Merge pull request #177 from NCAR/main_candidate_2021-02-19
gustavo-marques Feb 26, 2021
5667b70
Remove unused imports and delete blank line
gustavo-marques Feb 26, 2021
e97d229
Merge pull request #179 from gustavo-marques/flux_areacorr_eval
mvertens Feb 26, 2021
d8c0122
Merge branch 'pr1342' into merge_new_file_layout
alperaltuntas Mar 4, 2021
e87dffb
Merge remote-tracking branch 'gfdl/main' into merge_new_file_layout
alperaltuntas Mar 8, 2021
80cb48e
Merge pull request #181 from NCAR/merge_new_file_layout
gustavo-marques Mar 15, 2021
7ab7daa
updates and bugfixes for area correction factors
Mar 24, 2021
b8aaa88
changed computation of model_areas
Mar 24, 2021
973632c
calculate area correction factors only over non-zero mask values
Mar 25, 2021
1965f94
Merge branch 'dev/ncar' into fix_conflicts_flux_areacorr
gustavo-marques Mar 26, 2021
b5e9892
Merge pull request #183 from gustavo-marques/fix_conflicts_flux_areacorr
gustavo-marques Mar 26, 2021
a4bf344
add if clause to a set_diffusivity OMP block including clock calls.
alperaltuntas Mar 29, 2021
9e7324e
Merge pull request #184 from NCAR/fix_PET_test
alperaltuntas Mar 29, 2021
69c86cd
add support for threading in cmeps
jedwards4b Apr 5, 2021
2da6ee0
initialize localPet
jedwards4b Apr 5, 2021
8a8d9ea
response to review
jedwards4b Apr 9, 2021
c4c8747
Merge pull request #185 from jedwards4b/nuopc_threading
alperaltuntas Apr 9, 2021
0a34093
Merge branch 'main' into main-candidate-ncar-2021-04-21
gustavo-marques Apr 21, 2021
3b121cf
Replaces ESMF_LogFoundError to ChkErr
gustavo-marques Apr 21, 2021
6196c03
Add isPresent and isSet when retrieving nthreads
gustavo-marques Apr 22, 2021
e6ce6a8
use isPresent,isSet to conditionally retrieve configuration
gustavo-marques Apr 26, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
168 changes: 148 additions & 20 deletions config_src/drivers/nuopc_cap/mom_cap.F90
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,22 +35,25 @@ module MOM_cap_mod
use MOM_ocean_model_nuopc, only: ocean_model_init, update_ocean_model, ocean_model_end
use MOM_ocean_model_nuopc, only: get_ocean_grid, get_eps_omesh
use MOM_cap_time, only: AlarmInit
use MOM_cap_methods, only: mom_import, mom_export, mom_set_geomtype, state_diagnose
use MOM_cap_methods, only: mom_import, mom_export, mom_set_geomtype, mod2med_areacor
use MOM_cap_methods, only: med2mod_areacor, state_diagnose
use MOM_cap_methods, only: ChkErr

#ifdef CESMCOUPLED
use shr_file_mod, only: shr_file_setLogUnit, shr_file_getLogUnit
use shr_mpi_mod, only : shr_mpi_min, shr_mpi_max
#endif
use time_utils_mod, only: esmf2fms_time

use, intrinsic :: iso_fortran_env, only: output_unit

use ESMF, only: ESMF_ClockAdvance, ESMF_ClockGet, ESMF_ClockPrint
use ESMF, only: ESMF_ClockAdvance, ESMF_ClockGet, ESMF_ClockPrint, ESMF_VMget
use ESMF, only: ESMF_ClockGetAlarm, ESMF_ClockGetNextTime, ESMF_ClockAdvance
use ESMF, only: ESMF_ClockSet, ESMF_Clock, ESMF_GeomType_Flag, ESMF_LOGMSG_INFO
use ESMF, only: ESMF_Grid, ESMF_GridCreate, ESMF_GridAddCoord
use ESMF, only: ESMF_GridGetCoord, ESMF_GridAddItem, ESMF_GridGetItem
use ESMF, only: ESMF_GridComp, ESMF_GridCompSetEntryPoint, ESMF_GridCompGet
use ESMF, only: ESMF_LogFoundError, ESMF_LogWrite, ESMF_LogSetError
use ESMF, only: ESMF_LogWrite, ESMF_LogSetError
use ESMF, only: ESMF_LOGERR_PASSTHRU, ESMF_KIND_R8, ESMF_RC_VAL_WRONG
use ESMF, only: ESMF_GEOMTYPE_MESH, ESMF_GEOMTYPE_GRID, ESMF_SUCCESS
use ESMF, only: ESMF_METHOD_INITIALIZE, ESMF_MethodRemove, ESMF_State
Expand All @@ -69,9 +72,10 @@ module MOM_cap_mod
use ESMF, only: ESMF_COORDSYS_SPH_DEG, ESMF_GridCreate, ESMF_INDEX_DELOCAL
use ESMF, only: ESMF_MESHLOC_ELEMENT, ESMF_RC_VAL_OUTOFRANGE, ESMF_StateGet
use ESMF, only: ESMF_TimePrint, ESMF_AlarmSet, ESMF_FieldGet, ESMF_Array
use ESMF, only: ESMF_FieldRegridGetArea
use ESMF, only: ESMF_ArrayCreate
use ESMF, only: ESMF_RC_FILE_OPEN, ESMF_RC_FILE_READ, ESMF_RC_FILE_WRITE
use ESMF, only: ESMF_VMBroadcast
use ESMF, only: ESMF_VMBroadcast, ESMF_VMReduce, ESMF_REDUCE_MAX, ESMF_REDUCE_MIN
use ESMF, only: ESMF_AlarmCreate, ESMF_ClockGetAlarmList, ESMF_AlarmList_Flag
use ESMF, only: ESMF_AlarmGet, ESMF_AlarmIsCreated, ESMF_ALARMLIST_ALL, ESMF_AlarmIsEnabled
use ESMF, only: ESMF_STATEITEM_NOTFOUND, ESMF_FieldWrite
Expand All @@ -93,6 +97,7 @@ module MOM_cap_mod
use NUOPC_Model, only: model_label_SetRunClock => label_SetRunClock
use NUOPC_Model, only: model_label_Finalize => label_Finalize
use NUOPC_Model, only: SetVM
!$use omp_lib , only : omp_set_num_threads

implicit none; private

Expand Down Expand Up @@ -141,6 +146,7 @@ module MOM_cap_mod
integer :: scalar_field_count = 0
integer :: scalar_field_idx_grid_nx = 0
integer :: scalar_field_idx_grid_ny = 0
integer :: nthrds !< number of openmp threads per task
character(len=*),parameter :: u_FILE_u = &
__FILE__

Expand Down Expand Up @@ -412,10 +418,12 @@ subroutine InitializeAdvertise(gcomp, importState, exportState, clock, rc)
character(len=512) :: diro
character(len=512) :: logfile
character(ESMF_MAXSTR) :: cvalue
character(len=64) :: logmsg
logical :: isPresent, isPresentDiro, isPresentLogfile, isSet
logical :: existflag
integer :: userRc
integer :: localPet
integer :: localPeCount
integer :: iostat
integer :: readunit
character(len=512) :: restartfile ! Path/Name of restart file
Expand All @@ -440,7 +448,7 @@ subroutine InitializeAdvertise(gcomp, importState, exportState, clock, rc)
call ESMF_VMGetCurrent(vm, rc=rc)
if (ChkErr(rc,__LINE__,u_FILE_u)) return

call ESMF_VMGet(VM, mpiCommunicator=mpi_comm_mom, rc=rc)
call ESMF_VMGet(VM, mpiCommunicator=mpi_comm_mom, localPet=localPet, rc=rc)
if (ChkErr(rc,__LINE__,u_FILE_u)) return

call ESMF_ClockGet(CLOCK, currTIME=MyTime, TimeStep=TINT, RC=rc)
Expand All @@ -452,7 +460,30 @@ subroutine InitializeAdvertise(gcomp, importState, exportState, clock, rc)
CALL ESMF_TimeIntervalGet(TINT, S=DT_OCEAN, RC=rc)
if (ChkErr(rc,__LINE__,u_FILE_u)) return

!TODO: next two lines not present in NCAR
!---------------------------------
! openmp threads
!---------------------------------

call ESMF_VMGet(vm, pet=localPet, peCount=localPeCount, rc=rc)
if (ChkErr(rc,__LINE__,u_FILE_u)) return

if(localPeCount == 1) then
Copy link
Copy Markdown
Contributor

@DeniseWorthen DeniseWorthen Apr 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The two instances of retrieving the attribute 'nthreads' are failing for EMC. They will need "isPresent,isSet" conditionals added. Thanks.

Copy link
Copy Markdown
Collaborator Author

@gustavo-marques gustavo-marques Apr 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Thanks, @DeniseWorthen.

call NUOPC_CompAttributeGet(gcomp, name="nthreads", value=cvalue, &
isPresent=isPresent, isSet=isSet, rc=rc)
if (ChkErr(rc,__LINE__,u_FILE_u)) return
if (isPresent .and. isSet) then
read(cvalue,*) nthrds
else
nthrds = localPeCount
endif
else
nthrds = localPeCount
endif
write(logmsg,*) nthrds
call ESMF_LogWrite(trim(subname)//': nthreads = '//trim(logmsg), ESMF_LOGMSG_INFO)

!$ call omp_set_num_threads(nthrds)

call fms_init(mpi_comm_mom)
call constants_init
call field_manager_init
Expand Down Expand Up @@ -799,6 +830,7 @@ subroutine InitializeRealize(gcomp, importState, exportState, clock, rc)
integer :: lbnd3,ubnd3,lbnd4,ubnd4
integer :: nblocks_tot
logical :: found
logical :: isPresent, isSet
integer(ESMF_KIND_I4), pointer :: dataPtr_mask(:,:)
real(ESMF_KIND_R8), pointer :: dataPtr_area(:,:)
real(ESMF_KIND_R8), pointer :: dataPtr_xcen(:,:)
Expand All @@ -807,23 +839,37 @@ subroutine InitializeRealize(gcomp, importState, exportState, clock, rc)
real(ESMF_KIND_R8), pointer :: dataPtr_ycor(:,:)
integer :: mpicom
integer :: localPet
integer :: localPeCount
integer :: lsize
integer :: ig,jg, ni,nj,k
integer, allocatable :: gindex(:) ! global index space
character(len=128) :: fldname
character(len=256) :: cvalue
character(len=256) :: frmt ! format specifier for several error msgs
character(len=512) :: err_msg ! error messages
integer :: spatialDim
integer :: numOwnedElements
type(ESMF_Array) :: elemMaskArray
real(ESMF_KIND_R8) , pointer :: ownedElemCoords(:)
real(ESMF_KIND_R8) , pointer :: lat(:), latMesh(:)
real(ESMF_KIND_R8) , pointer :: lon(:), lonMesh(:)
integer(ESMF_KIND_I4) , pointer :: mask(:), maskMesh(:)
real(ESMF_KIND_R8) :: diff_lon, diff_lat
real :: eps_omesh
real(ESMF_KIND_R8) :: L2_to_rad2
type(ESMF_Field) :: lfield
real(ESMF_KIND_R8), allocatable :: mesh_areas(:)
real(ESMF_KIND_R8), allocatable :: model_areas(:)
real(ESMF_KIND_R8), pointer :: dataPtr_mesh_areas(:)
real(ESMF_KIND_R8) :: max_mod2med_areacor
real(ESMF_KIND_R8) :: max_med2mod_areacor
real(ESMF_KIND_R8) :: min_mod2med_areacor
real(ESMF_KIND_R8) :: min_med2mod_areacor
real(ESMF_KIND_R8) :: max_mod2med_areacor_glob
real(ESMF_KIND_R8) :: max_med2mod_areacor_glob
real(ESMF_KIND_R8) :: min_mod2med_areacor_glob
real(ESMF_KIND_R8) :: min_med2mod_areacor_glob
character(len=*), parameter :: subname='(MOM_cap:InitializeRealize)'
integer :: spatialDim
integer :: numOwnedElements
type(ESMF_Array) :: elemMaskArray
real(ESMF_KIND_R8) , pointer :: ownedElemCoords(:)
real(ESMF_KIND_R8) , pointer :: lat(:), latMesh(:)
real(ESMF_KIND_R8) , pointer :: lon(:), lonMesh(:)
integer(ESMF_KIND_I4) , pointer :: mask(:), maskMesh(:)
real(ESMF_KIND_R8) :: diff_lon, diff_lat
real :: eps_omesh
!--------------------------------

rc = ESMF_SUCCESS
Expand Down Expand Up @@ -851,6 +897,28 @@ subroutine InitializeRealize(gcomp, importState, exportState, clock, rc)
call ESMF_VMGet(vm, petCount=npet, mpiCommunicator=mpicom, localPet=localPet, rc=rc)
if (ChkErr(rc,__LINE__,u_FILE_u)) return

!---------------------------------
! openmp threads
!---------------------------------

call ESMF_VMGet(vm, pet=localPet, peCount=localPeCount, rc=rc)
if (ChkErr(rc,__LINE__,u_FILE_u)) return

if(localPeCount == 1) then
call NUOPC_CompAttributeGet(gcomp, name="nthreads", value=cvalue, &
isPresent=isPresent, isSet=isSet, rc=rc)
if (ChkErr(rc,__LINE__,u_FILE_u)) return
if (isPresent .and. isSet) then
read(cvalue,*) nthrds
else
nthrds = localPeCount
endif
else
nthrds = localPeCount
endif

!$ call omp_set_num_threads(nthrds)

!---------------------------------
! global mom grid size
!---------------------------------
Expand Down Expand Up @@ -992,17 +1060,76 @@ subroutine InitializeRealize(gcomp, importState, exportState, clock, rc)
end if
end do

deallocate(ownedElemCoords)
deallocate(lonMesh , lon )
deallocate(latMesh , lat )
deallocate(maskMesh, mask)
! realize the import and export fields using the mesh
call MOM_RealizeFields(importState, fldsToOcn_num, fldsToOcn, "Ocn import", mesh=Emesh, rc=rc)
if (ChkErr(rc,__LINE__,u_FILE_u)) return

call MOM_RealizeFields(exportState, fldsFrOcn_num, fldsFrOcn, "Ocn export", mesh=Emesh, rc=rc)
if (ChkErr(rc,__LINE__,u_FILE_u)) return

!---------------------------------
! determine flux area correction factors - module variables in mom_cap_methods
!---------------------------------
! Area correction factors are ONLY valid for meshes that are read in - so do not need them for
! grids that are calculated internally

! Determine mesh areas for regridding
call ESMF_MeshGet(Emesh, numOwnedElements=numOwnedElements, spatialDim=spatialDim, rc=rc)
if (ChkErr(rc,__LINE__,u_FILE_u)) return

allocate (mod2med_areacor(numOwnedElements))
allocate (med2mod_areacor(numOwnedElements))
mod2med_areacor(:) = 1._ESMF_KIND_R8
med2mod_areacor(:) = 1._ESMF_KIND_R8

#ifdef CESMCOUPLED
! Determine model areas and flux correction factors (module variables in mom_)
call ESMF_StateGet(exportState, itemName=trim(fldsFrOcn(2)%stdname), field=lfield, rc=rc)
if (ChkErr(rc,__LINE__,u_FILE_u)) return
call ESMF_FieldRegridGetArea(lfield, rc=rc)
if (ChkErr(rc,__LINE__,u_FILE_u)) return
call ESMF_FieldGet(lfield, farrayPtr=dataPtr_mesh_areas, rc=rc)
if (ChkErr(rc,__LINE__,u_FILE_u)) return

allocate(mesh_areas(numOwnedElements))
allocate(model_areas(numOwnedElements))
k = 0
do j = ocean_grid%jsc, ocean_grid%jec
do i = ocean_grid%isc, ocean_grid%iec
k = k + 1 ! Increment position within gindex
if (mask(k) /= 0) then
mesh_areas(k) = dataPtr_mesh_areas(k)
model_areas(k) = ocean_grid%AreaT(i,j) / ocean_grid%Rad_Earth**2
mod2med_areacor(k) = model_areas(k) / mesh_areas(k)
med2mod_areacor(k) = mesh_areas(k) / model_areas(k)
end if
end do
end do
deallocate(mesh_areas)
deallocate(model_areas)

! Write diagnostic output for correction factors
min_mod2med_areacor = minval(mod2med_areacor)
max_mod2med_areacor = maxval(mod2med_areacor)
min_med2mod_areacor = minval(med2mod_areacor)
max_med2mod_areacor = maxval(med2mod_areacor)
call shr_mpi_max(max_mod2med_areacor, max_mod2med_areacor_glob, mpicom)
call shr_mpi_min(min_mod2med_areacor, min_mod2med_areacor_glob, mpicom)
call shr_mpi_max(max_med2mod_areacor, max_med2mod_areacor_glob, mpicom)
call shr_mpi_min(min_med2mod_areacor, min_med2mod_areacor_glob, mpicom)
if (localPet == 0) then
write(logunit,'(2A,2g23.15,A )') trim(subname),' : min_mod2med_areacor, max_mod2med_areacor ',&
min_mod2med_areacor_glob, max_mod2med_areacor_glob, 'MOM6'
write(logunit,'(2A,2g23.15,A )') trim(subname),' : min_med2mod_areacor, max_med2mod_areacor ',&
min_med2mod_areacor_glob, max_med2mod_areacor_glob, 'MOM6'
end if
#endif

deallocate(ownedElemCoords)
deallocate(lonMesh , lon )
deallocate(latMesh , lat )
deallocate(maskMesh, mask)

else if (geomtype == ESMF_GEOMTYPE_GRID) then

!---------------------------------
Expand Down Expand Up @@ -1229,7 +1356,6 @@ subroutine InitializeRealize(gcomp, importState, exportState, clock, rc)

call MOM_RealizeFields(exportState, fldsFrOcn_num, fldsFrOcn, "Ocn export", grid=gridOut, rc=rc)
if (ChkErr(rc,__LINE__,u_FILE_u)) return

endif

!---------------------------------
Expand Down Expand Up @@ -1405,6 +1531,8 @@ subroutine ModelAdvance(gcomp, rc)

call shr_file_setLogUnit (logunit)

!$ call omp_set_num_threads(nthrds)

! query the Component for its clock, importState and exportState
call ESMF_GridCompGet(gcomp, clock=clock, importState=importState, &
exportState=exportState, rc=rc)
Expand Down
Loading