Introduce multi instance capability

[alperaltuntas]

This PR introduces the ability to run multiple instances of MOM6 in CESM. This is required for ensemble runs, and DART, in particular. The main changes are in buildnml to accomodate multiple MOM6 runtime parameter files, MOM_input, MOM_override, diag_table, and input.nml.

This PR should be evaluated in conjunction with NCAR/MOM6#241

[gustavo-marques]

@alperaltuntas; I run a test with NINST_OCN=4 and got only one file per diag stream (e.g., casename.mom6.sfc_0001_01.nc). Is this expected? I do see four copies of the log files and MOM input files...

[gustavo-marques]

I am still getting only one file per diag stream of MOM6 (e.g., casename.mom6.sfc_0001_01.nc).

[gustavo-marques]

Here is the case I ran:
/glade/scratch/gmarques/gmom.e23.GJRA.TL319_t061_zstar_N65.dev_ncar_04may23/run

[alperaltuntas]

Can you try it with FMS2?

./xmlchange MOM6_INFRA_API=FMS2

[gustavo-marques]

It worked with MOM6_INFRA_API=FMS2.

[klindsay28]

I'm now seeing that my mods for CESM filenames has a conflict with this that I'll need to resolve when I merge the new main to my branch. This makes me wonder if we should add a test of multi instance to aux_mom, to make sure that this capability doesn't get broken.

[klindsay28]

I'm running a multi instance case with my mods for CESM filenames to make sure that I'm not breaking it.
It looks like there is only 1 ocn log file, and it only has log output from 1 instance.
It looks like that was also the case with Gustavo's case,
RUNDIR=/glade/scratch/gmarques/gmom.e23.GJRA.TL319_t061_zstar_N65.dev_ncar_04may23/run
It looks like the other components, except cesm.log, have a log file per instance.
This seems like a gap in multi instance support. Should there be a separate issue to track this?

[alperaltuntas]

Interesting. I am getting multiple ocn log files:

/glade/scratch/altuntas/g.e23.G_JRA.TL319_t061.multi.012/run

I'll investigate why you only get one.

[klindsay28]

My case has now run for 2 months and is at
RUNDIR=/glade/scratch/klindsay/g.e23b12.GMOM_JRA.TL319_t061.003/run

Your case used cmeps0.14.2, while mine used cmeps0.14.24 and Gustavo's used cmeps0.14.14.
I haven't dug into the changes in CMEPS between these tags yet.