Provide tools and approaches for robust determination of pass/fail of CTest -S driver scripts

[bartlettroscoe]

CC: @fryeguy52, @jwillenbring, @william76

Description

CTest -S invocations will not always return zero when all of the requested actions pass. Sometimes, it will return nonzero, such as when build warnings are found or if no tests are run (see https://gitlab.kitware.com/snl/project-1/issues/54). In addition, CTest -S driver scripts are run in two primary situations, as described below, that need to be supported.

The first situation for running ctest -S scripts is to just put results up on CDash and the results of the build and tests will be grabbed from there. In that use case, we usually want to consider that the outer CTest -S process passes if it does not have any errors of its own and if it was able to submit results to CDash. This would be the case with most nightly testing builds. In this situation, we don't want to consider the outer CTest -S script as failing if the inner configure, build, or tests fail (since all of that info is sent to the project's CDash site). We only want to consider the outer ctest -S command as failed if there is an error in the ctest -S script that does not get posted to the project's CDash site. Various types of errors can occur in the outer ctest -S script that are not submitted to the inner project CDash site like problems generating some of the files that are sent up to the inner Project's CDash stie, problems with enable/disable logic in the other script, etc. By only reporting the other ctest -S script fails in cases like that, it will be easier to spot that failure (by looking on another CDash project or looking on a Jenkins site) since we expect the project's configure, build, and test to fail from time to time. But we don't expect the outer ctest -S script to ever fail and we want to be altered if it does since that is system infrastructure issue, not a project code or test issue.

The second situation for running ctest -S scripts is to both submit to CDash and determine pass/fail of the inner project in the same process that runs the ctest -S script. For example, in a Jenkins pipeline, we would like a way to determine if there were any failures both in the outer ctest -S script or in the inner project's configure, build or tests. In this case, any failure in the outer ctest -S script or the inner product's configure, build, or tests should be noted as a failure (but it would still be nice to distinquish the two types of failures).

One challenge here is that inside of the ctest -S driver script, you can't detect if a failure has occurred or not (see https://gitlab.kitware.com/snl/project-1/issues/52). For example, if ctest raises an error and prints "CTest Error", then it is currently (as of CMake/CTest 3.11) impossible for the script code to detect that. The detection of those errors can only be determined by grepping the STDERR output from the ctest -S driver script (which hits as to the implementation we need to follow below).

Another issue that we need to address is what is happening with Trilinos testing on one of the ATDM platforms where the bsub command (with the LSF scheduler) that runs the ctest -S command crashes and returns a zero error code, even though the ctest -S script did not finish building, testing, and submitting results to CDash! In cases like that, one can never rely on return codes.

All of these issues make it so that you can't rely on return codes from ctest -S, period.

This story is to set up some tools and approaches to support these various use cases that does not rely on the return code from ctest -S but yet can still accurately report pass/fail in a robust way.

Possible implementations

The are several possible approaches to achieve what we want (as described above).

First, note that implementing an option <Project>_CTEST_FAIL_ON_CONFIG_BUILD_TEST_FAILURE in TribitsCTestDriverCore.cmake (similar to what was mentioned in #154) would result in the TRIBITS_CTEST_DRIVER() script printing passed or failed based on if the update, configure, biuld, or tests failed or not. But, as mentioned above, that can't detect failures in ctest itself. Therefore, this is not a complete solution but it will be part of any solution that we implement.

One approach would be to try to get Kitware to update CTest so that it would return the correct error code for the situations described above. But that would require a forced upgrade of CMake/CTest everywhere and it would not be able to catch problems like when the bsub command crashes.

Another approach would be to write a python script that runs ctest -S and then scraps the STDOUT and STDERR output (as it echos it as well) and then determines pass/fail by greping the output. This would not require any updates to CMake or CTest and would give us full control of what "pass" means and what "fails" means. But this also would not catch problems like the bsub command crashing.

The last option considered here would be to simply tee the output of the ctest -S <script>.cmake command to a file and then grep the file to determinging pass/failure. This is similar to the last approach mentioned but now this script could run in the same process or a different process that runs the ctest -S <script>.cmake command. For example, the script that runs inside of the bsub command could run:

ctest -S <script>.name 2>&1 | tee ctest-s-driver.out

Then in the script that calls the bsub command could run:

tribits-ctest-driver-pass-fail.sh ctest-s-driver.out

And then in the script tribits-ctest-driver-pass-fail.sh we could define whatever logic we wanted to determine pass/fail. The TRIBITS_CTEST_DRIVER() function itself would either print or not print TRIBITS_CTEST_DRIVER: OVERALL: ALL PASSSED based on <Project>_CTEST_FAIL_ON_CONFIG_BUILD_TEST_FAILURE. The script tribits-ctest-driver-pass-fail.sh would print "PASSED" and return a zero return code if it found TRIBITS_CTEST_DRIVER: OVERALL: ALL PASSSED in the output but did not find any text suggesting an error like "CMake Error" or "Submit failed". This should provide a robust solution that works in all of the situations described above.

Tasks

1. Add option <Project>_CTEST_FAIL_ON_CONFIG_BUILD_TEST_FAILURE (default TRUE) to TribitsCTestDriverCore.cmake and unit tests for various use cases ...
2. Implement tribits-ctest-driver-pass-fail.sh with appropriate unit tests and system-level tests ...

[bartlettroscoe]

Here is an example where ctest -S output shows that everything passed with "=TRIBITS_CTEST_DRIVER: OVERALL: ALL PASSSED printed at the end and no "CMake Error" lines printed and yet Jenkins thinks the build failed:

• https://jenkins-srn.sandia.gov/view/Trilinos/job/Trilinos-atdm-sems-gcc-7-2-0/124/consoleText

Here is another example:

• https://jenkins-srn.sandia.gov/view/Trilinos%20ATDM/job/Trilinos-atdm-rhel6-gnu-opt-openmp/22/consoleText

In this case, note the message TRIBITS_CTEST_DRIVER: OVERALL: ALL PASSSED and no "CMake Error" lines but 'ctest -S ctest-drivers.cmake' command returned code '255'. In this case, Jenkins returns passed just because it always does because the last command was an echo statement that passed.

This is what we have to fix.