Explicit module loading

[marshallward]

A change to the Gaea module environment was corrupting the user's
environment in the MOM gitlab testing, causing all module commands to
fail and making it impossible to build and run the models.

This may be caused by the introduction of readonly variables in the
/etc/bash.bashrc.local file, which prevented critical modules from
being loaded (presumably to prevent double module load statements).
It is possible these were incorrectly inherited to noninteractive Gaea
sessions.

This patch introduces a more explicit module setup. The process is
described below in more detail, in case further changes are needed.

1. module purge

   Clear out any existing modules (but not the readonly flags)

2. source /opt/cray/pe/modules/default/etc/modules.sh

   Bootstrap the module system and add and following modulepaths:

   • /opt/modulefiles: Compilers
   • /opt/cray/modulefiles: Lower level filesystem
   • /opt/cray/pe/modulefiles: PrgEnv-* and libraries

3. Load the following modules:

   • craype-network-aries
   • eproxy
   • PrgEnv-${compiler}
   • craype-broadwell
   • cray-mpich

4. Select compiler version and load the version-dependent modules:

   • cray-hdf5
   • cray-netcdf

Much of this process replicates the steps in /etc/bash.bashrc.local.

Advantages to this approach:

• It works.
• It creates a more minimal and predictable environment.

Disadvantages:

• Paths to modules.sh and modulefiles are hard-coded and need updates.
• The list of required modules may change over time.

[marshallward]

I just noticed that /etc/bash.bashrc.local was updated again on July 14, moving those crucial modules outside of the readonly flag's if-block. So this PR is probably no longer needed.

But if we want to take charge of our modules, then we can still consider it.

[adcroft]

On the one hand, the testing was originally defined to only work on our production machine and replicate our production environment. On the other, controlling the modules might be amenable to porting our testing to other sites. However, given that we might not (probably won't) have modules at other sites, I'm not sure if this change helps us or not. I've said that the mrs merge was just a pre-cursor to a re-design but waiting until we get around to that might re-encounter the trigger for this PR. I'm happy to moved forward with this as is but if you think it's unnecessary then I'll leave it up to you.

[marshallward]

It's no more localized than anything else in tools/MRS, so I don't think we need to consider the portability. This is a GFDL fix to a GFDL problem.

The underlying problem was later fixed on Gaea's end, but I still think it's better to be explicit, and this explicitly sets our modules, so it is probably the better solution. It also inoculates us from future undocumented changes to /etc/bash.bashrc.local.

If there are future changes, I'd rather be made aware of them than have them implicitly changed again.

A downside here is that our test environment is now more controlled, and no longer reflects the usual managed environment of a user.

If we want a porcelain test environment, then we should do this. If we want a typical user environment, then we should probably not do it and should pester Ops if this happens again.

[marshallward]

We were unable to come to a consensus on this issue, mainly due to indifference. The original problem no longer exists, and in all likelihood won't happen again.

I'll keep this branch around in case we want to revisit explicit module control, but for now it's probably OK to close this PR.