Re-work testing verification, fail file naming consistency, remove lsf support, simplify RT logs.

[BrianCurtis-NOAA]

Commit Queue Requirements:

• Fill out all sections of this template.
• All sub component pull requests have been reviewed by their code managers.
• Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
• Commit 'test_changes.list' from previous step

---

Description

This PR updates the rt.sh script:

• Makes fail_test and fail_compile files consistent in naming
• Created a new generate_log() function in rt.sh to allow for better control over the information logged after the full RT suite is finished.
  • Log outputs minimized to information that's important to track, and easier to read for developers.
  • Adds test_changes.list file for repository tracking of what tests are changed due to the changes in the PR.
  • Adds test_changes.list file for easy creation of new baselines specific to the PR using rt.sh -b test_changes.out.
• Ends support for lsf (queue system from old WCOSS systems)

Commit Message

* UFSWM
  * Re-work test verification in rt.sh for developers to use in PR process. (Closes #2058)
  * fail_<test_number> file name upgraded to use compile/test name instead of number.
  * Regression test logs do not clearly indicate when a test fails to run. (Closes #1821)
  * New test_changes.list file:
    * Verification of what tests change for each PR.
    * Used with './rt.sh -c -b test_changes.list` to help create new baselines for each PR.

Priority

• Normal

Git Tracking

UFSWM

• Closes Re-work test verification and add pre-test option in rt.sh for developers to use in PR process. #2058
• Closes regression test logs do not clearly indicate when a test fails to run #1821

Sub component Pull Requests

• None

UFSWM Blocking Dependencies

• None

---

Changes

Regression Test Changes (Please commit test_changes.list):

• None Expected

Input data Changes

• None.

Library Changes/Upgrades

• No Updates

---

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • Jet
  • Gaea
  • Derecho
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
• opnReqTest (complete task if unnecessary)

[BrianCurtis-NOAA]

@zach1221 it looks like the Derecho issue would block any PR at this point, should we proceed with testing? Have the ORT's been successful?

I don't think there's another PR ready at the moment. We could continue testing against this PR today, on the other machines, while Derecho is worked on. I've got ORTs running on Hera now. @jkbk2004 do you want to wait until Derecho is ready?

@zach1221 can you submit jobs in sequential mode even if rocoto/eclfow are not functional ?

This would take an extremely long time.

[zach1221]

@zach1221 it looks like the Derecho issue would block any PR at this point, should we proceed with testing? Have the ORT's been successful?

I don't think there's another PR ready at the moment. We could continue testing against this PR today, on the other machines, while Derecho is worked on. I've got ORTs running on Hera now. @jkbk2004 do you want to wait until Derecho is ready?

@zach1221 can you submit jobs in sequential mode even if rocoto/eclfow are not functional ?

This would take an extremely long time.

It's still running since last night. Hopefully not too much longer.

[zach1221]

Jet is mostly done. Just had one test timeout that I'm rerunning.

[BrianCurtis-NOAA]

What changes were made in Derecho to allow these tests to pass? I didn't see any Derecho changes in that regard.

[zach1221]

What changes were made in Derecho to allow these tests to pass? I didn't see any Derecho changes in that regard.

I'm about to commit them.

[zach1221]

Ok we should be ready to proceed with final review here. @jkbk2004 fyi

[BrianCurtis-NOAA]

Should be good to go for review/merge.
@jkbk2004 @SadeghTabas-NOAA