Update rt.sh to allow creating new baselines only for a subset of tests

[DusanJovic-NOAA]

Description

A new option has been added to rt.sh script. If option -b <file> is passed to rt.sh together with -c (create new baselines) then the script will run only tests specified in <file> (one test name per line) and create new baselines which will be saved in ${NEW_BASELINE} directory. For all other tests specified rt.conf current baseline directory will be linked to ${NEW_BASELINE}.
This allows developers to create new baselines only for the tests they expect to change the baselines while using current baselines for tests they expect to not change the outputs.

For example:

If file named 'new_baselines' contain:

$ cat new_baselines
control_CubedSphereGrid
control_CubedSphereGrid_parallel

then, running:

./rt.sh -e -c -b new_baselines

wil create new baselines only for these two tests and put them in ${NEW_BASELINE}, while symlinks will be created for all other tests.

Finally, running:

./rt.sh -e -m

will run a full test in verify mode against new baselines, which will confirm that no other tests other than those listed above change the outputs.

Input data additions/changes

• No changes are expected to input data.
• Changes are expected to input data:
  • New input data.
  • Updated input data.

Anticipated changes to regression tests:

• No changes are expected to any regression test.
• Changes are expected to the following tests:

Subcomponents involved:

• AQM
• CDEPS
• CICE
• CMEPS
• CMakeModules
• FV3
• GOCART
• HYCOM
• MOM6
• NOAHMP
• WW3
• stochastic_physics
• none

Library Updates/Changes

• Not Needed
• Create separate issue in JCSDA/spack-stack asking for update to library. Include library name, library version.
• Add issue link from JCSDA/spack-stack following this item

Combined with PR's (If Applicable):

Commit Queue Checklist:

• Link PR's from all sub-components involved in section below
• Confirm reviews completed in ALL sub-component PR's
• Add all appropriate labels to this PR.
• Run full RT suite on either Hera/Cheyenne AND attach log to a PR comment.
• Add list of any failed regression tests to "Anticipated changes to regression tests" section.

Linked PR's and Issues:

Testing Day Checklist:

• This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR.
• Move new/updated input data on RDHPCS Hera and propagate input data changes to all supported systems.

Testing Log (for CM's):

• RDHPCS
  • Hera
  • Orion
  • Jet
  • Gaea
  • Cheyenne
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
  • Completed
• opnReqTest
  • N/A
  • Log attached to comment

[BrianCurtis-NOAA]

I would normally just cp rt.conf to rt.test and edit rt.test and run with -l. How does this improve on that?

[DusanJovic-NOAA]

I would normally just cp rt.conf to rt.test and edit rt.test and run with -l. How does this improve on that?

If you use '-l rt.test' in create mode (-c) you can not run full test afterwards verifying that all tests pass, some using new baselines, some using current baselines.

[BrianCurtis-NOAA]

I wonder if you could write something that upon baseline creation failure for a full rt suite, a file could be generated with the list of baseline generating tests that failed, making it easy to re-run the failed tests?

[DusanJovic-NOAA]

I wonder if you could write something that upon baseline creation failure for a full rt suite, a file could be generated with the list of baseline generating tests that failed, making it easy to re-run the failed tests?

I think this sentence:

This allows developers to create new baselines only for the tests they expect to change the baselines while using current baselines for tests they expect to not change the outputs.

describes that situation. How developers come up with the list of tests that need new baselines is up to them. Either by running full set of tests and finding which one failed or by reasoning based on the changes they made.

[zach1221]

Changes combined into #1467 . Closing