Remove provenance/model specific configuration

[jasonb5]

This PR removes provenance and model specific configuration from
CIME. CIME will now look under a models cime_config directory
for a directory named customize. All python files under this directory
will be loaded. Provence now works as a hook system, specific methods
need to be defined to enable provenance. Configuration flags that
alter how CIME operates may be defined here as well.

Provenance hooks:
save_build_provenance
save_prerun_provenance
save_postrun_provenance

Test suite: scripts_regression_tests
Test baseline: n/a
Test namelist changes: n/a
Test status: n/a

Fixes #3432 and #4153
User interface changes?: n
Update gh-pages html (Y/N)?: y

[billsacks]

I am removing myself as a reviewer since it's still at least 2 weeks before I'd get a chance to look at it and I don't want to hold it up. Thanks for your work on this @jasonb5 - this is a hugely valuable set of changes!

[jasonb5]

@jedwards4b I've addressed all the issues. I've attached the CESM configuration file. It should be place under cime_config\customize, see E3SM for an example. I'll update the wiki with instructions on adding new variables.

[jasonb5]

Here's the configuration file, cesm_config.zip.

[jedwards4b]

looks good - I ran scripts_regression_tests on cheyenne. There are a couple of failures I am looking into.

[jedwards4b]

One of the test failures was my fault, it's fixed with no changes. The other looks like it shouldn't run for cesm:

FAIL: test_save_timings_manual (test_sys_save_timings.TestSaveTimings)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/glade/work/jedwards/sandboxes/cesm2_x_alpha/cime/CIME/tests/test_sys_save_timings.py", line 74, in test_save_timings_manual
    self.simple_test(manual_timing=True)
  File "/glade/work/jedwards/sandboxes/cesm2_x_alpha/cime/CIME/tests/test_sys_save_timings.py", line 49, in simple_test
    self.run_cmd_assert_result(
  File "/glade/work/jedwards/sandboxes/cesm2_x_alpha/cime/CIME/tests/base.py", line 138, in run_cmd_assert_result
    self.assertEqual(stat, expected_stat, msg=msg)
AssertionError: 1 != 0 : 
    COMMAND: cd /glade/scratch/jedwards/scripts_regression_test.20220815_135956/SMS_Ln9_P1.f19_g16_rx1.A.cheyenne_intel.fake_testing_only_20220815_152750 && /glade/work/jedwards/sandboxes/cesm2_x_alpha/cime/CIME/Tools/save_provenance postrun
    FROM_DIR: /glade/work/jedwards/sandboxes/cesm2_x_alpha/cime
    SHOULD HAVE WORKED, INSTEAD GOT STAT 1
    OUTPUT: 
    ERRPUT: Traceback (most recent call last):
  File "/glade/work/jedwards/sandboxes/cesm2_x_alpha/cime/CIME/Tools/save_provenance", line 90, in <module>
    _main_func(__doc__)
  File "/glade/work/jedwards/sandboxes/cesm2_x_alpha/cime/CIME/Tools/save_provenance", line 83, in _main_func
    save_prerun_provenance(case, lid=lid)
NameError: name 'save_prerun_provenance' is not defined

a possible solution is:

diff --git a/CIME/tests/test_sys_save_timings.py b/CIME/tests/test_sys_save_timings.py
index ca3e0a61f..60ff05dc5 100644
--- a/CIME/tests/test_sys_save_timings.py
+++ b/CIME/tests/test_sys_save_timings.py
@@ -45,11 +45,11 @@ class TestSaveTimings(base.BaseTestCase):
 
         self.assertEqual(len(lids), 1, msg="Expected one LID, found %s" % lids)
 
-        if manual_timing:
-            self.run_cmd_assert_result(
-                "cd %s && %s/save_provenance postrun" % (casedir, self.TOOLS_DIR)
-            )
         if config.test_mode == "e3sm":
+            if manual_timing:
+                self.run_cmd_assert_result(
+                    "cd %s && %s/save_provenance postrun" % (casedir, self.TOOLS_DIR)
+                )
             provenance_glob = os.path.join(
                 timing_dir,
                 "performance_archive",

[jasonb5]

@jedwards4b I'll look into that error.

[jasonb5]

@jedwards4b I've fixed the issue, can you re-test please. Thanks!

[jedwards4b]

All scripts_regression_tests now pass with cesm on cheyenne.

[jasonb5]

@jedwards4b Thanks will merge this shortly.