Skip to content

Add UFS tracing instrumentation to nuopc/cmeps driver #1075

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
apcraig merged 19 commits into from
May 14, 2026
Merged

Add UFS tracing instrumentation to nuopc/cmeps driver #1075

Changes from 17 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
f124914
No ncat,ntrcr in call icepack_aggregate in ice_prescribed_mod.F90
NickSzapiro-NOAA Sep 27, 2024
5e57a89
Update to CICE-Consortium/main (2024-09-28)
NickSzapiro-NOAA Sep 30, 2024
cf01f33
Merge remote-tracking branch 'CICE-Consortium/main' into develop
NickSzapiro-NOAA Nov 18, 2024
513f8d1
Sync CICE to Consortium/main (2024-11)
NickSzapiro-NOAA Dec 3, 2024
ff7fd76
Enable writing restarts that can be triggered at any forecast time ("…
NickSzapiro-NOAA Dec 11, 2024
ea4582b
Merge 'CICE-Consortium/main' into EMC/develop
NickSzapiro-NOAA Jan 23, 2025
4411a15
Sync to CICE-Consortium/main (2025-01-23)
NickSzapiro-NOAA Feb 8, 2025
dce7176
Add ufs_tracing
DusanJovic-NOAA Apr 1, 2025
473f258
Merge main from CICE-Consortium
NickSzapiro-NOAA Jul 28, 2025
feb4c7b
Merge CICE-Consortium/main
NickSzapiro-NOAA Aug 6, 2025
90ed252
Merge CICE-Consortium/main (2025-08-06)
NickSzapiro-NOAA Aug 19, 2025
31921b1
Merge remote-tracking branch 'origin/develop' into ufs_tracing
DusanJovic-NOAA Aug 26, 2025
01024e6
Conditional compilation of ufs_tracing
DusanJovic-NOAA Sep 10, 2025
5278e39
Sync to CICE-Consortium 2025-11
NickSzapiro-NOAA Nov 19, 2025
c90ee5a
Sync to CICE-Consortium (2025-11-20)
NickSzapiro-NOAA Nov 25, 2025
f888f31
Merge remote-tracking branch 'origin/develop' into ufs_tracing
DusanJovic-NOAA Dec 5, 2025
d69a94d
Merge ufs_tracing from DusanJovic-NOAA
NickSzapiro-NOAA Dec 16, 2025
5671194
Dusan's Add ufs tracing instrumentation to nuopc/cmeps driver
NickSzapiro-NOAA May 12, 2026
ad16178
Merge CICE-Consortium/main
NickSzapiro-NOAA May 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions cicecore/drivers/nuopc/cmeps/ice_comp_nuopc.F90
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,9 @@ module ice_comp_nuopc
#ifndef CESMCOUPLED
use shr_is_restart_fh_mod, only : init_is_restart_fh, is_restart_fh, is_restart_fh_type
#endif
#ifdef UFS_TRACING
use ufs_trace_mod
#endif

implicit none
private
Expand Down Expand Up @@ -117,6 +120,7 @@ module ice_comp_nuopc
character(*), parameter :: u_FILE_u = &
__FILE__

integer :: mype = -1

@dabail10 dabail10 Dec 16, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is already my_task available in the code.

!=======================================================================
contains
!===============================================================================
Expand All @@ -128,12 +132,23 @@ subroutine SetServices(gcomp, rc)
integer, intent(out) :: rc

! Local variables
type(ESMF_VM) :: vm
character(len=*),parameter :: subname=trim(modName)//':(SetServices) '
!--------------------------------

rc = ESMF_SUCCESS
if (dbug > 5) call ESMF_LogWrite(subname//' called', ESMF_LOGMSG_INFO)

call ESMF_GridCompGet(gcomp, vm=vm,rc=rc)
if (ChkErr(rc,__LINE__,u_FILE_u)) return
call ESMF_VMGet(vm, localpet=mype, rc=rc)
if (ChkErr(rc,__LINE__,u_FILE_u)) return
Comment on lines +148 to +151

@anton-seaice anton-seaice Dec 16, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

    if (ChkErr(rc,__LINE__,u_FILE_u)) return
    call ESMF_VMGet(vm, localpet=mype, rc=rc)
    if (ChkErr(rc,__LINE__,u_FILE_u)) return

Are these used anywhere?

@NickSzapiro-NOAA NickSzapiro-NOAA Dec 16, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to set mype (before my_task and master_task are available in InitializeAdvertise)


#ifdef UFS_TRACING
if (mype == 0) call ufs_trace_init()

@dabail10 dabail10 Dec 16, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should remain consistent with the rest of the code and check:

if (my_task == master_task)

@NickSzapiro-NOAA NickSzapiro-NOAA Dec 16, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I thought the same. But, the tracing starts before my_task and master_task are available in InitializeAdvertise

@dabail10 dabail10 Dec 17, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we call init_grid1 before this phase?

@NickSzapiro-NOAA NickSzapiro-NOAA Dec 17, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe not (?) as I think call input_data has to happen before call init_grid1 like in cice_init1 to have all the namelist variables

But, it's more that conflicts with the intent to start the tracing as soon as possible in all subcomponents (before calling much else) towards getting the fullest timeline of the run. And for this to be "unobtrusive"

fwiw, I think master_task and mype==0 are really the same since

CICE/cicecore/cicedyn/infrastructure/comm/mpi/ice_communicate.F90

Line 84 in fa682b3

master_task = 0

@NickSzapiro-NOAA NickSzapiro-NOAA Dec 17, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess these lines could be moved up to SetServices from InitializeAdvertise if there is really strong preference

CICE/cicecore/drivers/nuopc/cmeps/ice_comp_nuopc.F90

Lines 346 to 368 in fa682b3

call ESMF_GridCompGet(gcomp, vm=vm, rc=rc)
if (ChkErr(rc,__LINE__,u_FILE_u)) return
call ESMF_VMGet(vm, mpiCommunicator=lmpicom, localPet=localPet, PetCount=npes, rc=rc)
if (ChkErr(rc,__LINE__,u_FILE_u)) return
#ifdef CESMCOUPLED
call ESMF_VMGet(vm, pet=localPet, peCount=nthrds, rc=rc)
if (ChkErr(rc,__LINE__,u_FILE_u)) return
if (nthrds==1) then
call NUOPC_CompAttributeGet(gcomp, "nthreads", value=cvalue, rc=rc)
if (ESMF_LogFoundError(rcToCheck=rc, msg=ESMF_LOGERR_PASSTHRU, line=__LINE__, file=u_FILE_u)) return
read(cvalue,*) nthrds
endif
!$ call omp_set_num_threads(nthrds)
#endif
!----------------------------------------------------------------------------
! Initialize cice communicators
!----------------------------------------------------------------------------
call init_communicate(lmpicom) ! initial setup for message passing
mastertask = .false.
if (my_task == master_task) mastertask = .true.

@dabail10 dabail10 Dec 17, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is not guaranteed on all systems that master_task = 0. Does the call to ufs_trace_init really have to happen at this phase? I realize there are often issues with circular dependencies. I feel like the tracing doesn't need to happen before the scatter and PE decomposition.

@NickSzapiro-NOAA NickSzapiro-NOAA Dec 17, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's just easier to trace fully than "be smart" about what needs to be traced

One example where accounting/separating the different phases was found to be very useful was in trying to reduce GFS initialization time (ufs-community/ufs-weather-model#2831). Like removing some advertised fields from FV3 reduced InitializeAdvertise (top to bottom)
image

if (mype == 0) call ufs_trace("cice", "SetServices", "B")
#endif

! the NUOPC gcomp component will register the generic methods
call NUOPC_CompDerive(gcomp, model_routine_SS, rc=rc)
if (ChkErr(rc,__LINE__,u_FILE_u)) return
Expand Down Expand Up @@ -169,6 +184,9 @@ subroutine SetServices(gcomp, rc)

if (dbug > 5) call ESMF_LogWrite(subname//' done', ESMF_LOGMSG_INFO)

#ifdef UFS_TRACING
if (mype == 0) call ufs_trace("cice", "SetServices", "E")
#endif
end subroutine SetServices

!===============================================================================
Expand All @@ -187,6 +205,9 @@ subroutine InitializeP0(gcomp, importState, exportState, clock, rc)
!--------------------------------

rc = ESMF_SUCCESS
#ifdef UFS_TRACING
if (mype == 0) call ufs_trace("cice", "InitializeP0", "B")
#endif

! Switch to IPDv01 by filtering all other phaseMap entries
call NUOPC_CompFilterPhaseMap(gcomp, ESMF_METHOD_INITIALIZE, &
Expand All @@ -201,6 +222,9 @@ subroutine InitializeP0(gcomp, importState, exportState, clock, rc)
write(logmsg,*) profile_memory
call ESMF_LogWrite('CICE_cap:ProfileMemory = '//trim(logmsg), ESMF_LOGMSG_INFO)

#ifdef UFS_TRACING
if (mype == 0) call ufs_trace("cice", "InitializeP0", "E")
#endif
end subroutine InitializeP0

!===============================================================================
Expand Down Expand Up @@ -256,6 +280,10 @@ subroutine InitializeAdvertise(gcomp, importState, exportState, clock, rc)
character(len=*), parameter :: subname=trim(modName)//':(InitializeAdvertise) '
!--------------------------------

#ifdef UFS_TRACING
if (mype == 0) call ufs_trace("cice", "InitializeAdvertise", "B")
#endif

call ufs_settimer(wtime)

call NUOPC_CompAttributeGet(gcomp, name="ScalarFieldName", value=cvalue, isPresent=isPresent, isSet=isSet, rc=rc)
Expand Down Expand Up @@ -766,6 +794,9 @@ subroutine InitializeAdvertise(gcomp, importState, exportState, clock, rc)

call t_stopf ('cice_init_total')
if (mastertask) call ufs_logtimer(nu_timer,msec,'InitializeAdvertise time: ',runtimelog,wtime)
#ifdef UFS_TRACING
if (mype == 0) call ufs_trace("cice", "InitializeAdvertise", "E")
#endif
end subroutine InitializeAdvertise

!===============================================================================
Expand Down Expand Up @@ -799,6 +830,10 @@ subroutine InitializeRealize(gcomp, importState, exportState, clock, rc)
!--------------------------------

rc = ESMF_SUCCESS
#ifdef UFS_TRACING
if (mype == 0) call ufs_trace("cice", "InitializeRealize", "B")
#endif

if (dbug > 5) call ESMF_LogWrite(subname//' called', ESMF_LOGMSG_INFO)

call ufs_settimer(wtime)
Expand Down Expand Up @@ -1000,6 +1035,9 @@ subroutine InitializeRealize(gcomp, importState, exportState, clock, rc)
call flush_fileunit(nu_diag)

if (mastertask) call ufs_logtimer(nu_timer,msec,'InitializeRealize time: ',runtimelog,wtime)
#ifdef UFS_TRACING
if (mype == 0) call ufs_trace("cice", "InitializeRealize", "E")
#endif
end subroutine InitializeRealize

!===============================================================================
Expand Down Expand Up @@ -1048,6 +1086,9 @@ subroutine ModelAdvance(gcomp, rc)
!--------------------------------

rc = ESMF_SUCCESS
#ifdef UFS_TRACING
if (mype == 0) call ufs_trace("cice", "ModelAdvance", "B")
#endif
if (mastertask) call ufs_logtimer(nu_timer,msec,'ModelAdvance time since last step: ',runtimelog,wtime)
call ufs_settimer(wtime)

Expand Down Expand Up @@ -1288,6 +1329,9 @@ subroutine ModelAdvance(gcomp, rc)
if (dbug > 5) call ESMF_LogWrite(subname//' done', ESMF_LOGMSG_INFO)

if (mastertask) call ufs_logtimer(nu_timer,msec,'ModelAdvance time: ',runtimelog,wtime)
#ifdef UFS_TRACING
if (mype == 0) call ufs_trace("cice", "ModelAdvance", "E")
#endif
call ufs_settimer(wtime)

end subroutine ModelAdvance
Expand Down Expand Up @@ -1442,6 +1486,9 @@ subroutine ModelFinalize(gcomp, rc)
!--------------------------------

rc = ESMF_SUCCESS
#ifdef UFS_TRACING
if (mype == 0) call ufs_trace("cice", "ModelFinalize", "B")
#endif
call ufs_settimer(wtime)
if (dbug > 5) call ESMF_LogWrite(subname//' called', ESMF_LOGMSG_INFO)
if (my_task == master_task) then
Expand All @@ -1452,6 +1499,9 @@ subroutine ModelFinalize(gcomp, rc)
if (dbug > 5) call ESMF_LogWrite(subname//' done', ESMF_LOGMSG_INFO)

if(mastertask) call ufs_logtimer(nu_timer,msec,'ModelFinalize time: ',runtimelog,wtime)
#ifdef UFS_TRACING
if (mype == 0) call ufs_trace("cice", "ModelFinalize", "E")
#endif

end subroutine ModelFinalize

Expand Down