Allow wave grid to be different than ocean/ice grid

[DeniseWorthen]

PR Checklist

• This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR. Please consult the ufs-weather-model wiki if you are unsure how to do this.

• This PR has been tested using a branch which is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR

• An Issue describing the work contained in this PR has been created either in the subcomponent(s) or in the ufs-weather-model. The Issue should be created in the repository that is most relevant to the changes in contained in the PR. The Issue and the dependent sub-component PR
  are specified below.

• Results for one or more of the regression tests change and the reasons for the changes are understood and explained below.

• New or updated input data is required by this PR. If checked, please work with the code managers to update input data sets on all platforms.

Instructions: All subsequent sections of text should be filled in as appropriate.

The information provided below allows the code managers to understand the changes relevant to this PR, whether those changes are in the ufs-weather-model repository or in a subcomponent repository. Ufs-weather-model code managers will use the information provided to add any applicable labels, assign reviewers and place it in the Commit Queue. Once the PR is in the Commit Queue, it is the PR owner's responsibility to keep the PR up-to-date with the develop branch of ufs-weather-model.

Description

Provide a detailed description of what this PR does. What bug does it fix, or what feature does it add? Is a change of answers expected from this PR? Are any library updates included in this PR (modulefiles etc.)?

Issue(s) addressed

• fixes Allow wave model to run on different grid than ocean and ice #1278

Testing

How were these changes tested? What compilers / HPCs was it tested with? Are the changes covered by regression tests? (If not, why? Do new tests need to be added?) Have regression tests and unit tests (utests) been run? On which platforms and with which compilers? (Note that unit tests can only be run on tier-1 platforms)

• hera.intel
• hera.gnu
• orion.intel
• cheyenne.intel
• cheyenne.gnu
• gaea.intel
• jet.intel
• wcoss2.intel
• wcoss_cray
• wcoss_dell_p3
• opnReqTest for newly added/changed feature
• CI

Dependencies

• waiting on add bilnr_nstod mapping between ocn/ice/waves NOAA-EMC/CMEPS#68

[DeniseWorthen]

I've done a test on cheyenne.intel to get an idea of how much difference a lower resolution wave model makes. I only changed the tripole resolution for the wave model, ocean and ice continue to run on the usual resolution.

For the cpld_control_c192_p8 test (fhmax=30), I subbed in the mx100 tripole mesh and mod_def from the cpld_control_p8 run directory. I obtained the following time comparisons:

c192mx050 : The total amount of wall time = 1477.676412
c192mx100 : The total amount of wall time = 1431.209095

A similar comparison for cpld_bmark_p8 (fhmax=6) using the mx050 tripole mesh and mod_def from the cpld_control_c192_p8 test:

c384mx025 : The total amount of wall time = 1691.170446
c384mx050 : The total amount of wall time = 1697.250803

[JessicaMeixner-NOAA]

@DeniseWorthen do you have run directories that I could look at by chance?

[DeniseWorthen]

I just ran the full RTs on hera; two hafs jobs failed (which I was not expecting), so my run directories still exist.

/scratch1/NCEPDEV/stmp2/Denise.Worthen/FV3_RT/rt_11753

You can copy from there and swap the mod_def and mesh that you want to use. The only configuration change you need to make is the name of the mesh file that the wave model uses in nems.configure.

[JessicaMeixner-NOAA]

Thanks @DeniseWorthen I tried to use your branch with the workflow and it failed, but I assume it was a set-up issue on my end. I'll check when I get a chance. I'm also curious if maybe the waves is not the slowest component on these tests and if maybe the wave model is still speeding up even if overall time is not. I'll copy some of your directories and tests my theory. Thanks for sharing the rundirs!

[JessicaMeixner-NOAA]

@DeniseWorthen I ran the ESMF profiler with the bencharmk and the c192 cases, I believe similar to what you ran and also found that the overall times were not huge changes, but looking at the wave model itself, there's huge speed ups (see /scratch1/NCEPDEV/stmp2/Jessica.Meixner/try_05/*/ESMF_Profile.summary)
For the cpld bmrk test, the wave model went from 380s -> 118s (0.25->0.5 wave model resolution) and for the C192 test 403s->112 s(0.5->1 resolution) so I do think this could help if the components had different resource usage.

I'm going to try some 5 day runs from the workflow now and will report back.

[DeniseWorthen]

@JessicaMeixner-NOAA OK, sounds good. I think we need to be careful though because the timesteps set in the mod-def might not be appropriate for the faster coupling (ie, bmark runs fast loop runs at 300 vs c192 at 600). I thought the ww3 global timestep was set to the coupling frequency?

[JessicaMeixner-NOAA]

@DeniseWorthen good point! I'll double check that now. The global time step does get over-written by the coupling time step, but likely might mean it's not optimal.

[DeniseWorthen]

I did had an issue on Hera.intel (NOAA-EMC/hpc-stack#470) with creating the mesh file for the rectilinear 30m grid, but I was finally able to do it using the following two commands (and reverting to intel 18).

ncremap -g glo_30m.SCRIP.nc -G latlon=321,720#lat_typ=cap#lat_drc=s2n#snwe=-80.0,80.0,-0.25,359.75
ESMF_Scrip2Unstruct glo_30m.SCRIP.nc mesh.glo_30m.nc 0

The center grid points for the SCRIP file are -80:80 and 0:359.5. I tested this w/ the bmark configuration and it is not working. I will update here when I know more.

[DeniseWorthen]

I corrected the SCRIP generation; it needs to be generated as a regional domain, not global when using NCO.

ncremap -g glo_30m.SCRIP.nc -G latlon=321,720#snwe=-80.25,80.25,-0.25,359.75#lat_typ=uni#lat_drc=s2n
ESMF_Scrip2Unstruct glo_30m.SCRIP.nc mesh.glo_30m.nc 0

I've verified w/ this mesh file, the model will at least run.

[DeniseWorthen]

Closed via #1256