DO NOT MERGE: Show TriBITS test failures with CMake 3.21.0 that don't occur with CMake 3.17.5 (#363) #394
Conversation
bartlettroscoe
force-pushed
the
363-github-actions-tests-show-ctest-3.21-failures
branch
2 times, most recently
from
ce8e62f to
90662cf
Compare
|
The CMake 3.21 failures compared against the identical CMake 3.17 passing tests are shown in the GHA build iteration shown here with results on CDash shown here: showing:
The failing tests for the cmake-3.21 build are shown at: showing:
These failures are broken down into two sets of failures of 5 tests in each set. The first set of 5 failures are those that show runs of ctest -S scripts that should be passing but otherwise return (slick "Show Matching Output") showing:
These are all cases where the ctest -S driver reports all passing and there is no hint in the output of anything failing but yet have the return code 255. If you compare those tests to the same tests with the cmake-3.17.5 build shown at: you can see that ctest -S returns 0 in all of those cases. The second set of 5 failing tests are shown in: showing:
These are all cases that have build errors on purpose. (The purpose of the tests is to ensure that |
|
Note that I was also able to manually reproduce these failures on my local RHEL 7 machine using a locally installed CMake 3.21.0 as described in #389 (comment) (see DETAILED NOTES in that comment). |
|
NOTE: I created: to get this on the SNL Kitware board. |
bartlettroscoe
force-pushed
the
363-github-actions-tests-show-ctest-3.21-failures
branch
2 times, most recently
from
82f3725 to
ca9aa79
Compare
|
NOTE: I took your advice and updated this branch to test all versions of CMake between 3.17 and 3.21 and it posted to CDash at: and showed the build and test summary:
and the failing tests are shown at: showing (sorted by build name then test name):
That shows that all of the TriBITS tests passed with CMake 3.18 but in CMake 3.19, one test:
started failing and then in CMake 3.21, the other 9 tests started failing. In those earlier versions, that test fails like shown for cmake 3.19 here showing: That same test scenario with cmake 3.17 shown here showed: Everything else is identical here. Just the CMake versions should be different. |
|
@bartlettroscoe superb Ross! I am happy, we could already benefit from that! |
…re failures (#363) This is to make it so that it is easy for Kitware to reproduce the CMake 3.21.0 issues. This commit should only exist on this side branch '363-github-actions-tests-show-ctest-3.21-failures' in a temp PR for testing purposes. I also commented out the disables of the tests that show the CMake 3.21 failures.
…well (#363) This should help show when CMake introduced the changes that broke these TriBITS tests.
|
It was determined in: that the majority of the test failures were due to a regression in CMake from 3.20 to 3.21 which was fixed on CMake 'mastesr' in: merged 2 weeks ago. The remainig failing test is which was first deployed in CMake 3.19.0 as shown at: I suspect this may not be a defect in CTest but might be a defect in this test. The test
But the package I submitted this case to CDash with results shown at: What is strange is that the configure failure for the It looks like those configure results in: just got dropped from CDash or something. What happened to them? In any case the next configure of the individual packages For the case of That is what it should show. But then why is there an attempt to build any targets? The build error shoiwn here shows: And we see something similar for the build error with the That is very interesting. So these are global targets that never get defined because the So this brings up an interesting problem. The outer The purpose of this test was to test the behavior of Instead, I propose that I change this test to accept this current behavior, document it, and kick the can down the road until a later time. Really no one is using the package-by-package build process anymore because it is too expensive. It is better to just keep everything working and configure, build, and test faster. |
|
I could fix TriBITS to make the test Here are my options:
So, what do I do here? I may need to think about this for a few minutes and come back to this later today. |
|
I think I have decided to go with option-2 above. I will not even bother to document the option I will implement that on this PR branch, test locally, and push to this PR branch so it can be tested with the different CMake version to verify. Then if that works out, I will cherry-pick that commit into a new branch and create a new PR to merge this (and undo the disable of this test for the CMake 3.21 GHA builds). |
bartlettroscoe
force-pushed
the
363-github-actions-tests-show-ctest-3.21-failures
branch
from
ca9aa79 to
3db1690
Compare
|
I just pushed commit 3db1690 which implements option-2 above that fixes the test showing:
Only the 9 other tests for CMake 3.21 still fail that will be fixed once CMake 3.21.2 is released and the GitHub Actions workflow is updated to use that version. I will now post a new PR for just the fixes in commit 3db1690 (along with re-enabling this test) so we can get that resolved. |
….19+ (TriBITSPub#363, TriBITSPub#394) I also added another test case for when libs are broken in required upstream package. See the detailed comments, especially in the file TribitsCTestDriverCoreHelpers.cmake.
…for CMake 3.21 (TriBITSPub#363, TriBITSPub#394) Now that TriBITS has a workaround for new CMake 3.19 behavior, this test passes. The rest of the tests can be enabled once CMake 3.21.2 is released and the GitHub Actions workflow is updated to use that version (see TriBITSPub#394).
|
I will now leave this in review, waiting for CMake 3.21.2 to release and then I will test with CMake 3.21.2 and close this out (like an issue). |
…tests-show-ctest-3.21-failures (#363, #394) I resolved conflicts in the following files: * cmake/ctest/github_actions/tribits_cmake-3.21.0_makefiles_python-3.8_g++-9_tweaks.cmake: * Removed the commented out fixed test * test/ctest_driver/TribitsExampleProject/CMakeLists.txt: * Took what is on 'master' * tribits/core/package_arch/TribitsGlobalMacros.cmake: * Took what is on 'master'
Also remove all of the test disables as cmake 3.21.2 has a patch that fixes all of the test failures.
Brings in numerous refactorings to TriBITS over the last 3 months, but there should be no breaks in backward compatibility. Almost every file in TriBITS is changed due to the lower-casing of command, macro and function names in PR TriBITSPub/TriBITS#379. But the main driver for this snapshot is to bring in the change in PR TriBITSPub/TriBITS#413 that should make it so that Kokkos INTERFACE_COMPILE_OPTIONS get propagated to downstream targets in TriBITS and therefore to external customers through installed <Package>Config.cmake files and IMPORTED targets. I should have done several snapshots in the last few months and not done a big snapshot like this (but I have been testing with Trilinos locally along the way). Overall, this merge brings in changes from a bunch of TriBITS PRs including (from most recent): * TriBITSPub/TriBITS#413: Change internal TriBITS target_link_libraries() to PUBLIC (TriBITSPub/TriBITS#299) component: core type: enhancement * TriBITSPub/TriBITS#410: Upgrade from cmake 3.21.0 to 3.21.2 (TriBITSPub/TriBITS#363, TriBITSPub/TriBITS#394) * TriBITSPub/TriBITS#394: DO NOT MERGE: Show TriBITS test failures with CMake 3.21.0 that don't occur with CMake 3.17.5 (TriBITSPub/TriBITS#363) * TriBITSPub/TriBITS#409: Add getTestDictStatusField() to handle empty 'status' field (SESW-383) component: ci_support type: enhancement * TriBITSPub/TriBITS#408: Deal with spaces in CDash url parts (SESW-383) component: ci_support type: enhancement * TriBITSPub/TriBITS#403: Spelling fixes * TriBITSPub/TriBITS#407: Move tribits_get_build_url_and_write_to_file() to stand-alone module (TriBITSPub/TriBITS#154) component: ctest_driver type: enhancement * TriBITSPub/TriBITS#388: Fixing typos (TriBITSPub/TriBITS#377) * TriBITSPub/TriBITS#406: Fix tribits_ctest_driver() package-by-package mode for CMake 3.19+ (TriBITSPub/TriBITS#363, TriBITSPub/TriBITS#394) component: ctest_driver type: bug * TriBITSPub/TriBITS#401: Improve GitHub Actions and CDash integration (TriBITSPub/TriBITS#154) type: enhancement * TriBITSPub/TriBITS#366: CI: draft action yaml * TriBITSPub/TriBITS#402: Revert some incorrect uppercase->lowercase changes * TriBITSPub/TriBITS#387: Build and deploy TriBITS documentation with Sphinx (TriBITSPub/TriBITS#386) component: documentation type: enhancement * TriBITSPub/TriBITS#398: Deal with pr null in not preprending build name (TriBITSPub/TriBITS#363) type: bug * TriBITSPub/TriBITS#396: Send PR results to 'Pull Request' CDash group and add PR ID to build names (TriBITSPub/TriBITS#363) type: enhancement * TriBITSPub/TriBITS#397: Print the ninja path and version (TriBITSPub/TriBITS#363) * TriBITSPub/TriBITS#393: GitHub Actions based testing for TriBITS (TriBITSPub/TriBITS#363) type: enhancement * TriBITSPub/TriBITS#389: TriBITS CI testing with GitHub Actions (TriBITSPub/TriBITS#363) type: enhancement * TriBITSPub/TriBITS#392: Fix broken tests for non-Fortran and CMake 3.21 builds (TriBITSPub/TriBITS#363) component: core * TriBITSPub/TriBITS#391: Fix up build_docs.sh for Sphinx doc generation (TriBITSPub/TriBITS#386) component: documentation type: enhancement * TriBITSPub/TriBITS#390: Add test for doc generation and fix usage of Python3 component: documentation type: bug * TriBITSPub/TriBITS#385: Replace last few references to TribitsDevelopersGuide.html (TriBITSPub/TriBITS#381) component: documentation type: enhancement * TriBITSPub/TriBITS#384: Split TribitsDevelopersGuide.* into TribitsUsersGuide.* and TribitsMaintainersGuide.* (TriBITSPub/TriBITS#381) component: documentation type: enhancement * TriBITSPub/TriBITS#383: Remove endfunction(<string>) and endmacro(<string>) (TriBITSPub/TriBITS#274, TriBITSPub/TriBITS#382) component: common_tpls type: bug * TriBITSPub/TriBITS#380: More package-arch data-structure documentation updates (TriBITSPub/TriBITS#63) component: documentation type: enhancement * TriBITSPub/TriBITS#379: Convert TriBITS to lower-case CMake command, macro, and function names (TriBITSPub/TriBITS#274) The following files were conflicting where I went with what is on the Trilinos 'develop' branch: * cmake/tribits/common_tpls/FindTPLBLAS.cmake * cmake/tribits/common_tpls/FindTPLLAPACK.cmake * cmake/tribits/common_tpls/FindTPLNetcdf.cmake (It looks like the above changes never made it back into TriBITS proper. The conflicts were due to the case changes in cmake command calls in these files due to TriBITSPub/TriBITS#379.) There was also a conflict in the file: * cmake/tribits/core/installation/TribitsProjectConfigTemplate.cmake.in I looked at that one carefully and I think that may have been due to fixes on both sides and then the case changes from TriBITSPub/TriBITS#379.
This PR is used to demonstrate the different behavior of CMake 3.21.0 vs. CMake 3.17.5 with the TriBITS test suite. It uses a GitHub Actions driver that posts to CDash two otherwise identical builds that differ only in the CMake version that they use. These test failures are blocking the deployment of the GitHub Actions testing in PR #393 to implement #363.