Move machine-based options from config.base to host files

[DavidHuber-NOAA]

Description

This moves all machine-specific options to the workflow/hosts files from the config.* files.

This also turns HPSS archiving on for WCOSS2 by default.

Resolves #2942
Resolves #3087

Type of change

• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? NO

How has this been tested?

CI testing on Hercules. The tracker and genesis jobs did not run (as expected).
WCOSS2 testing still needs to be performed.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have documented my code, including function, input, and output descriptions
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added

[DavidHuber-NOAA]

This will break the C96_atm3DVar_extended test case until a fix is made for the gfs_downstream.tar file (as mentioned in #3019).

[DavidHuber-NOAA]

There is an issue with the gfs_bufrsnd job that is preventing it from creating the output gfs_collective BUFR files. I am re-disabling HPSS archiving on WCOSS2 for the time being and will add this information to #3019.

Marking ready for review.

[TerrenceMcGuinness-NOAA]

This is a commendable PR and a key demonstration of how to refactor an "if-y diff-y" system specific configuration construct into a more centralized and pythonic approach.  I particularly liked how you enhanced the formation of the defaulting overrides.

[WalterKolczynski-NOAA]

Minor things and one thing to sneak in (or leave for another time).

[WalterKolczynski-NOAA]

WCOSS is failing for a couple reasons:

1. GDAS fails to build. I believe this is a known issue
2. A failure to create experiments:

Running create_experiment.py for 5 cases
C48_ATM
The create_experiment command (./create_experiment.py -y ../ci/cases/pr/C48_ATM.yaml --overwrite) failed with a non-zero status.  Output:
Traceback (most recent call last):
  File "./create_experiment.py", line 33, in <module>
    from wxflow import AttrDict, parse_j2yaml, Logger, logit
  File "/lfs/h2/emc/ptmp/walter.kolczynski/PR/PR_3053/global-workflow/workflow/wxflow/__init__.py", line 13, in <module>
    from .jinja import Jinja
  File "/lfs/h2/emc/ptmp/walter.kolczynski/PR/PR_3053/global-workflow/workflow/wxflow/jinja.py", line 16, in <module>
    @jinja2.pass_eval_context
AttributeError: module 'jinja2' has no attribute 'pass_eval_context'

Seems like the WCOSS upgrade changed something with jinja.

CC @aerorahul

[DavidHuber-NOAA]

Updated the WCOSS2 module files to load Python 3.8.6 into the environment. This should resolve the errors seen during CI testing and issue #3087.

[WalterKolczynski-NOAA]

CI Tests set up to run in /lfs/h2/emc/ptmp/walter.kolczynski/PR/PR_3053/RUNTESTS on WCOSS

[aerorahul]

looks good. Thanks for including the fix to the jinja2 error on wcoss2

[WalterKolczynski-NOAA]

CI Tests set up to run in /lfs/h2/emc/ptmp/walter.kolczynski/PR/PR_3053/RUNTESTS on WCOSS

[WalterKolczynski-NOAA]

CI Tests set up to run in /lfs/h2/emc/ptmp/walter.kolczynski/PR/PR_3053/RUNTESTS on WCOSS

[WalterKolczynski-NOAA]

C96C48_ufs_hybatmDA is failing because the gdas build fails due to issues with oops. I believe this is a known issue, but the test should be disabled in the meantime.

Other active tests pass.

[DavidHuber-NOAA]

I disabled the UFS DA tests for WCOSS2 as part of this PR. Thanks for the heads up.

[aerorahul]

@DavidHuber-NOAA
Do you have confidence that merging this is OK. You have run offline tests on WCOSS2, correct?

[DavidHuber-NOAA]

Yes, I am confident in this PR. Here is a summary of testing done on WCOSS2:

• I ran the setup scripts on WCOSS2 yesterday, and just now, to confirm the correct Python version is being used and that the aerosol input data paths are being set correctly.
• I successfully ran CI tests on November 1st-2nd.
• Walter ran CI tests yesterday (11/12). All tests passed except the UFS DA test, as (temporarily) expected.

[aerorahul]

@RussTreadon-NOAA
If you are in agreement that disabling ufs DA tests are acceptable, please let us know. We would like to merge this PR.

[CoryMartin-NOAA]

I have a hunch (but no evidence) that #3124 may be related to this?