Add an option to calculate bottom pressure from initialization for SAL

[herrwang0]

As of now, self-attraction and loading (SAL) needs an external file for reference bottom pressure. This PR adds a second option to calculate the reference bottom pressure based on the initial ocean state.

A new subroutine find_col_mass is added in MOM_interface_heights. The subroutine packs the code for calculating diagnostics col_mass and pbo.

A new runtime parameter SAL_REF_PBOT_CONFIG is added with two available options: file and init.

• The default file option reads variable REF_PBOT_VARNAME from file REF_PBOT_FILE.
• The new init option calculates bottom pressure using subroutine find_col_mass. When this option is used, the calculated bottom pressure is saved as variable REF_PBOT_VARNAME in file REF_PBOT_FILE, which can be used for restart runs.
• The init option is reserved only for new runs. If init option is used by a restart run, a warning is issued, the file option is instead logged, and the model will look for file REF_PBOT_FILE. The consideration here is to avoid changing parameters for a run done in segments.

There are two commits in this PR:

418438c: A new subroutine find_col_mass is added in MOM_interface_heights, which is used for col_mass and pbo diagnostics.
380f1b0: Added an init option to calculate reference bottom pressure for SAL.

[Hallberg-NOAA]

I have examined the code revisions, and they look correct to me (apart from one spelling error where "initialization" is spelled as "initialiaztion" on line 239 of MOM_self_attr_load.F90), but obviously we have a problem with the TC testing giving a series of errors. My suggestion is to fix this spelling error and see whether the TC testing errors go away on their own. If they do not, we will need to figure out why the TC testing is not working.

[herrwang0]

Thanks for reviewing the PR @Hallberg-NOAA . What puzzles me is that all the Mac tests passed. And also, all TC tests that I ran on Gaea C6 passed as well.

For what it worths, I retriggered the TC tests (by closing and re-opening the PR) earlier, but got the same results.

I wonder if it is related to the halo of h. Column mass calculation calls subroutine int_density_dz, which requires halo=1, in order to calculate horizontal integrals intx_dpa and inty_dpa. Two potential issues:

1. I am not sure if h's halo is up-to-date by the time these diagnostics are calculated, which may somehow in certain cases lead issue reported here?
2. The extra halo is obviously unnecessary if all we need is dp for calculating column mass. (but given that this only happens for diagnostics and initialization, it probably only trivially affects efficiency. )

Modifying int_density_dz with an optional input argument halo_size may be worth testing, but then again, I will have to find a way to recreate the problem offline...

[marshallward]

Thanks for reviewing the PR @Hallberg-NOAA . What puzzles me is that all the Mac tests passed. And also, all TC tests that I ran on Gaea C6 passed as well.

@herrwang0 The MacOS tests are passing because we disabled floating point exceptions some time ago, due to an error in the HDF5 library (which I believe has since been fixed). If those were re-enabled, then I believe MacOS would also fail.

When you run this manually, the default flags are very simple and do not enable floating point exceptions. This something we turn on for GitHub Actions. If you add FCFLAGS_DEBUG = -g -O0 -ffpe-trap=invalid,zero,overflow to your config.mk then you should be able to replicate it.

(If using Intel, then I believe the flag is -fpe0)

[herrwang0]

The bug is now identified and fixed thanks to the help from @marshallward .

[Hallberg-NOAA]

With the revisions to correct the bug that was showing up in TC testing, this PR is now looking to me like it is ready to be merged in.

[Hallberg-NOAA]

This PR has passed pipeline testing at https://gitlab.gfdl.noaa.gov/ogrp/mom6ci/MOM6/-/pipelines/28092 with the expected warnings about updated parameter descriptions in some MOM_parameter_doc files.