Restore repro across restarts and layout in generic tracer runs with OBC

[yichengt900]

#520 has successfully addressed layout and restart reproducibility concerns in physics-only runs with OBC, resulting in a runtime increase of approximately 3% in regional cases. Building on this progress, the PR further resolves reproducibility issues for generic tracer runs with OBC, ensuring tracer values at OBC are correctly read from restart files.

We also added a new RT case based on Northwest Atlantic configuration with generic CFC tracers, NWA12.CFC, to the CEFI-regional-MOM6 repro. The RT case consists of three steps::

1. A 2 hrs run with a 16x8 layout
2. A 1hr run with a 16x8 layout
3. A 1hr run with an 8x8 layout, with restart from 2.

The restarts from steps 1 and 3 are compared to ensure they are bitwise identical.

It's important to note that this PR will change answers when OBC is used. Additionally, further investigation into the impact of stencil size changes (currently, the stencil size is only increased when OBC is used) may be necessary.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 37.14%. Comparing base (744dbcc) to head (17a600b).

Additional details and impacted files

@@             Coverage Diff              @@
##           dev/gfdl     #577      +/-   ##
============================================
- Coverage     37.14%   37.14%   -0.01%     
============================================
  Files           271      271              
  Lines         80726    80727       +1     
  Branches      15061    15061              
============================================
  Hits          29988    29988              
- Misses        45141    45142       +1     
  Partials       5597     5597              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[marshallward]

Am I right that this is meant to replace #520? @kshedstrom ?

[yichengt900]

This is a clean version of #520 with an additional fix for generic tracers. I believe adding @kshedstrom as a co-author would be appropriate since the majority of changes are based on her PR. @kshedstrom, please feel free to chime in with any thoughts.

[marshallward]

OK thanks, just making sure before closing #520. @Hallberg-NOAA does this address the halo issue that you raised in #520?

[marshallward]

It looks like we will process #520 first and then handle this one. (It may be simpler to just submit any changes to @kshedstrom's #520 branch?)

[yichengt900]

@marshallward, thanks for the updates. I am fine with either option. @kshedstrom, please let me know your preference.

[kshedstrom]

I tried this branch and it passed a layout test, but not a restart test. I need to investigate more. Unfortunately, chinook is crazy busy these days.

[kshedstrom]

My tests pass now, not sure what happened before.

[marshallward]

#520 has been merged, so this should be updated accordingly.

[yichengt900]

sync with dev/gfdl, clean up history. Branch Should be updated now.

[marshallward]

Gaea regression: https://gitlab.gfdl.noaa.gov/ogrp/MOM6/-/pipelines/22618 ✔️