Remove FMS from UFS MPAS Atmosphere + More MPAS infrastructure in UFS

[dustinswales]

Description

This PR removes all FMS dependencies in the MPAS enabled UFS.

For logging and error handling, we now use the MPAS native logging functionality.

ESMF has replaced FMS for clock management. Most of the FMS functionality is included with ESMF, so this was straightforward.

With the FMS removal, we had to create some new infrastructure to provide MPAS configurability. This was necessary since we relied on FMS for runtime configurability.
A new routine, ufs_mpas_read_stream_lists, allows for runtime control over which fields are included in the various MPAS output streams (e.g., output, restart, diag, etc...). This provides the same functionality of standalone MPAS.

Initial tracer/constituent handling. This replaces the FMS tracer_tables. This may be revamped in the future to handle constituents differently.

New, non-FMS, handling of the atmospheric namelist.

Other things (non-FMS related) included here:

Added new interval to write restart files at different interval than output files. These are controlled by the UFS model_configure file.

Adopted shared UFS tool to control restart, and output, file frequency, shr_is_restart_fh_mod. This is used by other UFS components NUOPC caps. Thanks @NickSzapiro-NOAA

Output/restart file format now matches with standalone MPAS-A. Timestamp in file name was HH:MM:SS, changed to HH.MM.SS

Issue(s) addressed

• fixes UWM #3181

Testing

The MPAS output file format has changed, which results in error during the comparison step.

[NickSzapiro-NOAA]

I think history output and restart writes are separate and independent concerns and can be handled similarly or not.

At least currently, history seems more component-native as it's tied to customizable fields, instantaneous vs. averages vs. extrema, internal timestepping,...

Restart writes are more checkpoints so that each component and coupled model can restart and reproduce bit-for-bit (as operational requirement)

All we need is a simple utility to create an array of ESMF Times given restart_fh and a start time. No?

Yes, so let's have a shared restart_fh utility (whatever it's called and wherever it lives) that carries details (like referencing from startTime or setting it in model_configure because reading a list from ufs.configure was buggy...). Especially if the details can change, it's cleaner for different components to share it.

imo, it's ATM that insists on being particular with a separate "encoded" restart_interval in model_configure where "3 6 9" are hours, "3 -1" is a frequency, and "time" changes if ATM is doing IAU.

[dustinswales]

@NickSzapiro-NOAA Sounds good.
Just so we are on the same page.
The plan is that I will modify the restart_fh in the ATM cap to use the common code in CDEPS-interface?
Then within the MPAS atmosphere, we can use this utility to control all of the MPAS output streams (history, restart, diag, da_state)?

One comment though. output_fh is specific to the ATM, but how it's translated to ESMF times is identical to restart_fh. So there is no reason to not use this common utility for all output files?

[NickSzapiro-NOAA]

Maybe we're in the process of writing the page...

Do you want all of the MPAS streams (like history, restart, diag, da_state) to have to write at the same times? I suspect not.
ufsatm_cap keeps separate output_fh and restart_fh

I think it's important that restart_fh be unified across UFS. output_fh not so much.

If you wanted it all to work similarly, the current restart_fh utility would need to be more flexible.
Currently, the shared restart_fh utility has this underlying data structure:

type :: is_restart_fh_type
    logical :: write_restartfh = .false.
    type(ESMF_Time), allocatable :: restartFhTimes(:)
  end type is_restart_fh_type

and components then do some flavor of

! initialize from restart_fh attribute in model_configure:
#ifndef CESMCOUPLED
       type(is_restart_fh_type)     :: restartfh_info     ! For flexible restarts in UFS
       call ESMF_TimeIntervalGet( dtimestep, s=dtime, rc=rc )
       if (ChkErr(rc,__LINE__,u_FILE_u)) return
       call init_is_restart_fh(mcurrTime, dtime, my_task == master_task, restartfh_info)

! check every timestep:
call is_restart_fh(clock, restartfh_info, write_restartfh)
if (write_restartfh) force_restart_now = .true.

One option would be for ATM to have seperate objects type(is_restart_fh_type) :: history_info, restart_info, diag_info, da_state_info and change init_is_restart_fh to accept an optional ESMF_ConfigGetAttribute label to read so each can be set and checked independently. Like:

call init_is_restart_fh(mcurrTime, dtime, my_task == master_task, history_info, label='history_fh')
call init_is_restart_fh(mcurrTime, dtime, my_task == master_task, restart_info, label='restart_fh')
call init_is_restart_fh(mcurrTime, dtime, my_task == master_task, diag_info, label='diag_fh')
call init_is_restart_fh(mcurrTime, dtime, my_task == master_task, da_state_info, label='da_state_fh')

[NickSzapiro-NOAA]

Wouldn't you rather use the native MPAS streams to handle output?
https://www2.mmm.ucar.edu/projects/mpas/site/documentation/users_guide/configuring_io.html

[dustinswales]

Maybe we're in the process of writing the page...

Do you want all of the MPAS streams (like history, restart, diag, da_state) to have to write at the same times? I suspect not. ufsatm_cap keeps separate output_fh and restart_fh

I think it's important that restart_fh be unified across UFS. output_fh not so much.

If you wanted it all to work similarly, the current restart_fh utility would need to be more flexible. Currently, the shared restart_fh utility has this underlying data structure:

type :: is_restart_fh_type
    logical :: write_restartfh = .false.
    type(ESMF_Time), allocatable :: restartFhTimes(:)
  end type is_restart_fh_type

and components then do some flavor of

! initialize from restart_fh attribute in model_configure:
#ifndef CESMCOUPLED
       type(is_restart_fh_type)     :: restartfh_info     ! For flexible restarts in UFS
       call ESMF_TimeIntervalGet( dtimestep, s=dtime, rc=rc )
       if (ChkErr(rc,__LINE__,u_FILE_u)) return
       call init_is_restart_fh(mcurrTime, dtime, my_task == master_task, restartfh_info)

! check every timestep:
call is_restart_fh(clock, restartfh_info, write_restartfh)
if (write_restartfh) force_restart_now = .true.

One option would be for ATM to have seperate objects type(is_restart_fh_type) :: history_info, restart_info, diag_info, da_state_info and change init_is_restart_fh to accept an optional ESMF_ConfigGetAttribute label to read so each can be set and checked independently. Like:

call init_is_restart_fh(mcurrTime, dtime, my_task == master_task, history_info, label='history_fh')
call init_is_restart_fh(mcurrTime, dtime, my_task == master_task, restart_info, label='restart_fh')
call init_is_restart_fh(mcurrTime, dtime, my_task == master_task, diag_info, label='diag_fh')
call init_is_restart_fh(mcurrTime, dtime, my_task == master_task, da_state_info, label='da_state_fh')

We are on the same page.
My plan was to have an instance of type(is_restart_fh_type) for each MPAS stream.

[dustinswales]

Wouldn't you rather use the native MPAS streams to handle output? https://www2.mmm.ucar.edu/projects/mpas/site/documentation/users_guide/configuring_io.html

We do...

The only thing I need output_fh, restart_fg, xyz_fh, is to derive the ESMF Time used to trigger the output calls.

Currently, I take output_fh and restart_fh and translate them to ESMF time arrays.
Then these are used to trigger output into the MPAS streams after forward integration. These output calls use the MPAS native IO.
Each MPAS stream should have its own trigger.

What I don't like about this is that I have to convert from and a real array into the ESMF times. This could be set in the NUOPC cap using the utility you suggested, by having multiple instances of is_restart_fh_type for each MPAS stream.

[dustinswales]

@NickSzapiro-NOAA Here's some more history and how this fits into this effort.

When bringing the MPAS dynamical core into the UFS, we had to draw a line of responsibility between the host (UFS) and the dycore (MPAS-A). MPAS-A is a standalone model with its own infrastructure, and we only want to use its dynamical core. MPAS-A has infrastructure for I/O (i.e, mpas_write_stream and mpas_read_stream) and additional infrastructure for controlling the I/O (i.e. MPAS_stream_manager). We drew the responsibility line in-between. So we are using MPAS native I/O, but the control of I/O is the hosts (UFS) responsibility.

This responsibility on the host is modest. We need to tell the MPAS atmosphere (a) when to write which stream (e.g. output_fh, restart_fh, etc...) and (b) which fields to include in these streams (e.g. stream_list.atmosphere.output)

The PR merged yesterday added the MPAS output stream, using output_fh the derive the ESMF times needed to trigger the output stream. No functionality for a stream_list included here, so all available fields are included in the output file.

This PR does the same thing, but for the MPAS restart stream. Additionally, this PR adds infrastructure to use the MPAS stream_list files to control which fields are included in the stream. If no stream_list file is provided, all available fields in the stream are written (previous basic functionality).

Our next PR will include the physics-to-dynamics coupling, which will introduce another MPAS stream (diag) to the UFS. So having a common way for the host (UFS) to communicate the requested output file times for the various MPAS streams is much needed.

[dustinswales]

@NickSzapiro-NOAA In the last commit I reverted my changes and adopted using shr_is_restart_fh_mod for determining the MPAS output stream times.
I had to extend shr_is_restart_fh_mod to use it for other stream types, but the default is to use restart_fh, so it is backward compatible.

[gspetro-NOAA]

Could we get reviews on this PR? We'd like to process its parent PR (ufs-community/ufs-weather-model#3201) next.

[gspetro-NOAA]

Testing for ufs-community/ufs-weather-model#3201 is complete. This PR can be merged.

[NickSzapiro-NOAA]

fyi, build_ufsatm CI now fails as

CMake Error at CMakeLists.txt:179 (add_library):
  Cannot find source file:

    /home/runner/work/ufsatm/ufsatm/ufsatm/CDEPS-interface/ufs/cdeps_share/shr_is_restart_fh_mod.F90

  Tried extensions .c .C .c++ .cc .cpp .cxx .cu .mpp .m .M .mm .ixx .cppm
  .ccm .cxxm .c++m .h .hh .h++ .hm .hpp .hxx .in .txx .f .F .for .f77 .f90
  .f95 .f03 .hip .ispc


CMake Error at CMakeLists.txt:179 (add_library):
  No SOURCES given to target: ufsatm_fv3


CMake Generate step failed.  Build files cannot be regenerated correctly.

[DusanJovic-NOAA]

fyi, build_ufsatm CI now fails as

CMake Error at CMakeLists.txt:179 (add_library):
  Cannot find source file:

    /home/runner/work/ufsatm/ufsatm/ufsatm/CDEPS-interface/ufs/cdeps_share/shr_is_restart_fh_mod.F90

  Tried extensions .c .C .c++ .cc .cpp .cxx .cu .mpp .m .M .mm .ixx .cppm
  .ccm .cxxm .c++m .h .hh .h++ .hm .hpp .hxx .in .txx .f .F .for .f77 .f90
  .f95 .f03 .hip .ispc


CMake Error at CMakeLists.txt:179 (add_library):
  No SOURCES given to target: ufsatm_fv3


CMake Generate step failed.  Build files cannot be regenerated correctly.

I suspected this might happen now that we introduced a dependency on the source code from outside of this repo.

[dustinswales]

@NickSzapiro-NOAA The offending file is not in UFSATM/, it's in ufs-weather-model/
This path:
/home/runner/work/ufsatm/ufsatm/ufsatm/CDEPS-interface/ufs/cdeps_share/shr_is_restart_fh_mod.F90?
I believe should be,
/home/runner/work/ufsatm/ufsatm/../CDEPS-interface/ufs/cdeps_share/shr_is_restart_fh_mod.F90?

I

[DusanJovic-NOAA]

@NickSzapiro-NOAA The offending file is not in UFSATM/, it's in ufs-weather-model/ This path: /home/runner/work/ufsatm/ufsatm/ufsatm/CDEPS-interface/ufs/cdeps_share/shr_is_restart_fh_mod.F90? I believe should be, /home/runner/work/ufsatm/ufsatm/../CDEPS-interface/ufs/cdeps_share/shr_is_restart_fh_mod.F90?

I

Let's merge this PR as it is, and we'll try to find solution and fix CI in next PR