WRF Coop em_real Tests Using SANE Workflows

[islas]

This PR introduces a set of tests that allows replication of the WRF Coop Tests which are normally run as regression tests for PRs.

TYPE: enhancement

KEYWORDS: testing, cicd, continuous integration

SOURCE: internal

DESCRIPTION OF CHANGES:
Problem:
The current regression tests found in the WRF Coop repository suffer from a few key design points:

1. located in a separate repository allowing code divergence and extra maintenance burden
2. confusing layout due to multiple repositories and data file locations
3. test logic obfuscation due to actual code to be executed auto-generated
4. limited execution tightly coupled to a containerized environment

PR #2095 tried to remedy this using hpc-workflows, however the framework likewise suffered from issues:

1. manual unconventional environment management
2. duplication of effort between tests and lack of support for dependencies between common actions (e.g. re-using builds across multiple tests)
3. limited support for extensibility outside of argument manipulation

Solution:
This PR does not aim to entirely replace PR #2095 (notably the CI/CD GitHub worklow) and instead leverages this point in PR #2095:

3. Write CI/CD tests in a simple and generally CI/CD framework-agnostic method where definitions of these also reside within the WRF repo

These tests follow this same mantra of "CI/CD framework-agnostic" such that they can more or less be a drop in replacement only for the hpc-workflows-based tests.

The tests will cover the WRF Coop Test Cases (provided is a default configuration for Derecho):

Tests
em_real	em_realG
em_realA	em_realH
em_realB	em_realI
em_realC	em_realJ
em_realD	em_realK
em_realE	em_realL
em_realF	various build tests

The tests are now written in the SANE Workflows framework, which solves most of the issues faced by the other two setups. Data is still spread across multiple locations, but that is separate from the testing code.

The structure of the tests is as follows:

.sane/                          #< The root directory in WRF where the testing code is kept
└── wrf                         #< A subfolder to make all python-imports look like `import wrf`
    ├── custom_actions
    │   └── run_wrf.py          #< A module that has our custom reusable classes to setup initial conditions and model runs
    ├── hosts
    │   ├── derecho_envs.jsonc  #< The environments that derecho.jsonc has
    │   └── derecho.jsonc       #< Definition of derecho HPC system for this framework
    ├── scripts                 #< A subfolder to house all our shell helper scripts that do the bulk of the work
    │   ├── buildCMake.sh
    │   ├── buildMake.sh
    │   ├── compare_wrf.sh      #< Use diffwrf to compare two runs
    │   ├── run_init.sh         #< Configurable to run initial conditions (em_real.exe or ideal.exe)
    │   ├── run_wrf_restart.sh  #< Runs wrf.exe again in previous run folder and compares history
    │   └── run_wrf.sh          #< Runs wrf.exe
    └── tests                   #< Where our tests live
        ├── builds
        │   └── builds.py       #< Python module that sets up ALL our compilation tests (make + cmake)
        └── regtests
            └── wrf_coop.py     #< Python module that sets up the WRF Coop em_real* tests

Documentation for this new framework can be found at:
https://sane-workflows.readthedocs.io/en/latest/

One could run these tests on Derecho using the following commands (inside a WRF repo clone):

python3 -m venv .venv/wrf_testing
source .venv/wrf_testing/bin/activate
python3 -m pip install --pre sane-workflows
# Runs the em_real test case
sane_runner --path .sane/ --actions em_real --run

[weiwangncar]

The regression test results:

Test Type              | Expected  | Received |  Failed
= = = = = = = = = = = = = = = = = = = = = = = =  = = = =
Number of Tests        : 23           24
Number of Builds       : 60           57
Number of Simulations  : 158           150        0
Number of Comparisons  : 95           86        0

Failed Simulations are: 
None
Which comparisons are not bit-for-bit: 
None

[weiwangncar]

@islas I get an error running source .venv/wrf_testing/bin/activate: Badly placed ()'s. Should I run activate.csh since I'm using tcsh?

[islas]

@weiwangncar Yes, sorry, the instructions assumed some use of sh shell

[weiwangncar]

Works as the current capability allows.