Accommodate multi-instance runs in CESM

[alperaltuntas]

This PR introduces changes that are mainly in NUOPC cap, infra/FMS, and src/framework to accommodate multi-instance runs in CESM:

Changes in nuopc cap:

• receive instance id and suffix from the CESM coupler.
• call ensemble_manager_init. This makes sure the input/output files contain the instance suffix.
• add instance suffix to log files, restart files, and rpointer.

Changes in infra/FMS{1,2}:

• add the optional ensumble_suffix arg to ensemble_manager_init. If provided, this arg bypasses the call to FMS_ensemble_manager_init, and instead, calls fms_io_set_filename_appendix and fms2_io_set_filename_appendix. This makes sure that all i/o files managed my FMS contains the ensemble suffix.

Change in src/framework:

• append instance suffix to MOM_parameter_doc* files. ( I believe this is the only change that may potentially impact GFDL's ensemble workflow.)

This PR should be evaluating in conjunction with ESCOMP/MOM_interface#137.

I am mentioning @MJHarrison-GFDL, @marshallward, and @hkershaw-brown in case they'd prefer to review this PR before it gets merged to dev/ncar. If so, please let us know and will hold off until you review.

[codecov-commenter]

Codecov Report

Patch coverage: 31.49% and project coverage change: +1.30 🎉

Comparison is base (59cc9ed) 35.88% compared to head (7344fcd) 37.18%.

❗ Current head 7344fcd differs from pull request most recent head fb10ddc. Consider uploading reports for the commit fb10ddc to get more accurate results

Additional details and impacted files

@@             Coverage Diff              @@
##           dev/ncar     #241      +/-   ##
============================================
+ Coverage     35.88%   37.18%   +1.30%     
============================================
  Files           260      263       +3     
  Lines         72088    73207    +1119     
  Branches      13601    13621      +20     
============================================
+ Hits          25868    27223    +1355     
+ Misses        41171    40976     -195     
+ Partials       5049     5008      -41     

Impacted Files	Coverage Δ
src/core/MOM_unit_tests.F90	16.66% <ø> (ø)
src/framework/MOM_get_input.F90	62.16% <0.00%> (ø)
src/tracer/MOM_neutral_diffusion.F90	65.38% <ø> (ø)
src/tracer/MOM_tracer_types.F90	0.00% <ø> (ø)
src/framework/MOM_file_parser.F90	90.45% <25.00%> (+30.25%)	⬆️
src/tracer/MOM_tracer_hor_diff.F90	76.51% <25.00%> (+0.03%)	⬆️
src/tracer/MOM_hor_bnd_diffusion.F90	61.87% <29.41%> (ø)
src/tracer/MOM_tracer_registry.F90	64.72% <33.33%> (+0.18%)	⬆️
...nfig_src/infra/FMS1/MOM_ensemble_manager_infra.F90	30.00% <75.00%> (+2.22%)	⬆️
src/core/MOM.F90	51.18% <100.00%> (ø)
... and 1 more

... and 9 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[marshallward]

The changes to MOM and the FMS infra should be fine. I have a pending PR which removes the references to fms_io_mod in the FMS2 ensemble manager, but that conflict should not be difficult to resolve when it happens.

(I would suggest looking into squashing and rebasing some of these commits though!)

[hkershaw-brown]

I ran an 8 instance case with this, looks good.

[alperaltuntas]

@gustavo-marques , I just realized there is a conflict due to the latest PR merge. I'll address that.

[alperaltuntas]

@gustavo-marques this is ready to merge. You can squash and merge. To run with multiple instances you can set ./xmlchange NINST_OCN=N where N is the number of MOM6 instances you'd like to run with.