Improve error checking in rt.sh

[DusanJovic-NOAA]

Commit Queue Requirements:

• Fill out all sections of this template.
• All sub component pull requests have been reviewed by their code managers.
• Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
• Commit 'test_changes.list' from previous step

---

Description:

Update rt scripts to fix checking the job exit status from Slurm and PBS schedulers. Fix checking the return code from check_results function, code from check_results is now inlined in run_test.sh. Add new test in error-test.conf to check for job wall clock timeout error. See #2379 for details.

Commit Message:

* UFSWM - Update rt scripts to fix checking the job exit status from Slurm and PBS schedulers.

Priority:

• Normal

Git Tracking

UFSWM:

• Closes Incorrect job status returned from squeue in rt.sh #2379

Sub component Pull Requests:

• None

---

Changes

Regression Test Changes (Please commit test_changes.list):

• No Baseline Changes.

Input data Changes:

• None.

Library Changes/Upgrades:

• No Updates

---

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • Jet
  • Gaea
  • Derecho
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
• opnReqTest (complete task if unnecessary)

[DusanJovic-NOAA]

Regression test passed on Hera. RegressionTests_hera.log

[BrianCurtis-NOAA]

I tested this on WCOSS2 with #2183 and the full suite passed. Since there were no errors, I can't verify how it performs if there was one. At least, though, it doesn't cause issue with a no-error RT suite.

[jkbk2004]

merged with #2389