Improve TribitsCTestDriverCore.cmake for integration with GitHub CI drivers #154

bartlettroscoe · 2016-11-10T20:35:49Z

Description

Currently, tribits_ctest_driver() just prints out the site name, the build name, and the URL of the CDash site. That is good but we can do better. We need to improve the URL to make it a single unique click. We can do that by:

Get “Build Stamp” from TAG file (which has <date-time> and <build-type>) to build CDash URL query with “Site”, “Build Name” and “Build Stamp” (as <date-time>-<build-type>):
Print out the updated CDASH URL at beginning and end of ctest -S driver output
Print out the updated CDash URL at the end of ‘make dashboard’ output

What the above will do is make it easy for people looking at the Travis CI output to find the detailed configure, build, and test info on CDash even as the builds and tests are occurring (and especially after the CI build is complete). Printing out the CDash URL at the end of make dashboard makes it easy for people running that command to find the output on CDash.

This must include extending the automated tests for tribits_ctest_driver() using TribitsExampleProject to check all of this.

The text was updated successfully, but these errors were encountered:

…#154) This is getting ready for the work to add more automated testing involving TribitsExampleProject and other projects. This gets the configure arugments for compilers, compiler options, etc that need to be passed down.

#154) This demonstrates and protects running the checking-test.py script directly from the standard location: <projectDir>/cmake/tribits/ci_support/checkin-test.py This is needed for Trilinos, for example.

bartlettroscoe · 2017-09-18T18:46:50Z

For example, the URL will look like:

https://testing.sandia.gov/cdash/index.php?project=Trilinos&display=project&filtercount=3&showfilters=1&filtercombine=and&field1=site&compare1=61&value1=crf450.srn.sandia.gov&field2=buildname&compare2=61&value2=Linux-GCC-4.8.4-MPI_RELEASE_DEBUG_SHARED_PT_CI_AAO&field3=buildstamp&compare3=61&value3=20170918-0700-Experimental

and that can be generated directly in the CTest -S driver script (i.e. in TribitsCTestDriverCore.cmake).

This is not the right location. Need to implement #154.

…cmake (TriBITSPub#278, TriBITSPub#154) This is needed to be able to read XML files out of the <bulid>/Testing/<buildstarttime>/ directory needed for (TriBITSPub#278) This will also make it easy to generate a CDash URL without knowing the build ID.

…tronger testing (TriBITSPub#278, TRIL-260) This commit represents a fairly significant enhancement of the handing of git repo clones and updates. Previous to this commit, CDash would not report the correct git repo SHA1 if the branch was not the default branch on the initial clone or in checking out a new branch. Also, the git commands were not robust in use cases where the current branch in a base repo or an extra repo was not on a tracking branch or the branch in the remote repo did not exist (like occurred in TRIL-260 which changing from 'atdm-develop-nightly' to 'atdm-nightly' where 'atdm-develop-nightly' got deleted in the GitHub repo). The documentation for git repo updates and been improved and streamlined. This includes a huge step forward in automated testing of the tribits_ctest_driver() function in how it does real git repo clones and updates for various use cases. To accomplish this, a new set of repos were create under https://github.com/tribits/ for the example projects/repos TribitsExampleMetaProject, TribitsExampleProject and TribitsExampleProjectAddons. There is a branch 'for-testing' in each of those repos that have known stats that are used for strong testing. Tests are set up based on a base TribitsExampleMetaProject repo/project that clones the extra repos TribitsExampleProject and TribitsExampleProjectAddons so we cover use cases in the base repo and extra repos (which are mostly decoupled). This testing helped to catch invalid logic for the extra repos (for the case where the current branch is not a tracking branch). Some of the specific things done include: * Switched from CTEST_UPDATE_COMMAND to CTEST_GIT_COMMAND (this really is git only) * Added CTEST_GIT_UPDATE_CUSTOM which calls a ctest -P script to do the improved git repo cleanup and update commands (see updated documentation). * Allow a set of CTEST_NOTES_FILES be specified by the user and correctly added to other notes files created internally and (hopefully) submitted to CDash. There are now strong unit tests for the full listing of notes files that are generated (which requires running ctest_update() to generate a few of the files). * Added TRIBITS_READ_CTEST_TAG_FILE() and TribitsGetCTestTestXmlDir.cmake (TriBITSPub#278, TriBITSPub#154): This is needed to be able to read XML files out of the <bulid>/Testing/<buildstarttime>/ directory needed for (TriBITSPub#278). This will also make it easy to generate a CDash URL without knowing the build ID. But this is not hooked in yet. It was added here with the hope to check the contents of the Notes.xml file but that file does not get generated unless you actually submit to CDash (which we can't do yet and don't want to rely on). Also added the file Add list_cdash_notes_files_in_xml_file.sh in hopes of doing this testing. * Remove the function to get the extra repo tracking branch. The old code assumed that the current branch in an extra repo was on a tacking branch. That is a bad assumption. The updated code just uses '@{u}' as the tracking branch. This was a real defect that was found through careful automated testing with real git commands. This is why we need some tests with real git commands to validate our implementation. * Updated some files in the example repos TribitsExampleMetaProject, TribitsExampleProject and TribitsExampleProjectAddons some to get these testing use cases working. Also improved the README files some more, but more is needed.

bartlettroscoe · 2020-05-05T19:36:06Z

Tasks to get the full CDash URL created.

Add correct URL to CDash for output results:
- Create function tribits_get_build_stamp_from_parts() to construct the buildstamp from buildstarttime and grup and write unit test ...
- Create function tribits_get_cdash_build_url_from_parts() based on site, buildname, and bulidstamp and add unit tests ...
- Write the CDash URL to the file <build>/CDashBuildUrl.txt ...
- Update existing test for ctest -S driver to check that the CDash URL is printed in STDOUT and contained in the file CDashBuildUrl.txt ...
- Update existing test for 'make dashboard' target to print the contents of CDashBuildUrl.txt at the end of the command ...

Can turn this on for things like GitHub Actions and other processes that take care of the repo versioning. This allows you to call ctest_update() and provide a link back to the commit on GitHub but not have ctest do any update action. ToDo: Add documentation!

…ON (TriBITSPub#154)

Version of CMake 3.17 (and likely earlier) added the model in addition to group to the TAG file.

I will squash this comit with commits that complete the system-level test for a build-specific URL (TriBITSPub#154).

…ITSPub#154) I will amend/squash this commit with the full implementation.

…riBITSPub#154) Need this for calling tribits_get_cdash_build_url_from_tag_file() from tribits_ctest_driver().

…riBITSPub#154)

Version of CMake 3.17 (and likely earlier) added the model in addition to group to the TAG file.

…riBITSPub#154) Need this for calling tribits_get_cdash_build_url_from_tag_file() from tribits_ctest_driver().

…riBITSPub#154)

…interaction Improve GitHub Actions and CDash integration (#154) * Links are provided from GitHub Actions Jobs to results on CDash (through CDash URL implemented manually in tribits_ctest_drdiver()) * Links on CDash builds are provided back to the GitHub PRs (through setting CTEST_CHANGE_ID=<pr-id>) * The builds on CDash are prefixed by the PR ID with pr_<pr-id>_ so CDash queries can be easily constructed that show all of these. * The Git repo SHA1 is shown on CDash (which shows merge commits for PRs but shows commits on 'master' for 'Continuous' and 'Nightly' builds).

bartlettroscoe · 2021-08-11T03:05:21Z

With the merge of PR #401, I think this is done. Note the detailed review of what was accomplished in #401 (comment) which is:

Links are provided from GitHub Actions Jobs to results on CDash (through CDash URL implemented manually in tribits_ctest_drdiver())
Links on CDash builds are provided back to the GitHub PRs (through setting CTEST_CHANGE_ID=<pr-id>)
The builds on CDash are prefixed by the PR ID with pr_<pr-id>_ so CDash queries can be easily constructed that show all of these.
The Git repo SHA1 is shown on CDash (which shows merge commits for PRs but shows commits on 'master' for 'Continuous' and 'Nightly' builds).

bartlettroscoe · 2021-08-11T17:01:02Z

FYI: I added the section GitHub Actions Pull Request Testing to the wiki to explain what is currently set up.

What I realized is missing is direct links on CDash back to the GitHub Action Job that ran it. It is not that hard to match them up based on the time stamps but this is something that is missing with the current system. If we wanted to provide links from CDash builds to the exact GitHub Actions job that run it, it seems that info might be provided by one of the env vars listed at:

https://docs.github.com/en/actions/reference/environment-variables

That might be future work to cap that off if I decide to write a report on this or something. (Being able to link from CDash to the exact job that ran it is missing on most systems that I have seen.)

I think this is ready to close.

bartlettroscoe · 2021-08-12T22:58:15Z

I just updated the section GitHub Actions Pull Request Testing on the wiki to show the link from GHA to CDash when the job fails. It is safe to close now.

…riBITSPub#154) Now the module TribitsGetCDashUrlFromTagFile.cmake contains everytthin (other than a call to tribits_read_ctest_tag_file() the module TribitsReadTagFile.cmake). I also refactored TribitsReadTagFile.cmake to not have any dependencies at all. Therefore, you can now just copy these two modules into your CMake project and use them with your ctest -S script. You can't get better than that.

…riBITSPub#154) Now the module TribitsGetCDashUrlFromTagFile.cmake contains everything (other than a call to tribits_read_ctest_tag_file() the module TribitsReadTagFile.cmake). I also refactored TribitsReadTagFile.cmake to not have any dependencies at all. Therefore, you can now just copy these two modules into your CMake project and use them with your ctest -S script. You can't get better than that.

…-module.cmake Move tribits_get_build_url_and_write_to_file() to stand-alone module (#154)

bartlettroscoe · 2021-08-16T17:23:57Z

Looking at how Kitware provides links from their GitLab Merge Requests to CDash, they use CDash queries based on the "Revision" a shown from this comment:

https://gitlab.kitware.com/cmake/cmake/-/merge_requests/6435

showing:

to this CDash query:

https://open.cdash.org/index.php?project=CMake&filtercount=1&field1=revision&compare1=61&value1=79d175b7aee8a8f58b98fe0d81c66f2cb7d68aef

showing:

I just realized that we could do the same thing for the TriBITS-provided CDash link for all of the PR builds for given PR testing iteration a shown in this query:

https://testing.sandia.gov/cdash/index.php?project=TriBITS&date=2021-08-13&filtercount=2&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=pr-403_&field2=revision&compare2=61&value2=6dec30cc351662a1fabf99c491ac36a2f3b3190e

showing:

That means that we could provide a URL to a CDash query for all of the builds associated with that GHA PR build. You could provide that in each ctest -S output in another hyperlink and another file.

Brings in numerous refactorings to TriBITS over the last 3 months, but there should be no breaks in backward compatibility. Almost every file in TriBITS is changed due to the lower-casing of command, macro and function names in PR TriBITSPub/TriBITS#379. But the main driver for this snapshot is to bring in the change in PR TriBITSPub/TriBITS#413 that should make it so that Kokkos INTERFACE_COMPILE_OPTIONS get propagated to downstream targets in TriBITS and therefore to external customers through installed <Package>Config.cmake files and IMPORTED targets. I should have done several snapshots in the last few months and not done a big snapshot like this (but I have been testing with Trilinos locally along the way). Overall, this merge brings in changes from a bunch of TriBITS PRs including (from most recent): * TriBITSPub/TriBITS#413: Change internal TriBITS target_link_libraries() to PUBLIC (TriBITSPub/TriBITS#299) component: core type: enhancement * TriBITSPub/TriBITS#410: Upgrade from cmake 3.21.0 to 3.21.2 (TriBITSPub/TriBITS#363, TriBITSPub/TriBITS#394) * TriBITSPub/TriBITS#394: DO NOT MERGE: Show TriBITS test failures with CMake 3.21.0 that don't occur with CMake 3.17.5 (TriBITSPub/TriBITS#363) * TriBITSPub/TriBITS#409: Add getTestDictStatusField() to handle empty 'status' field (SESW-383) component: ci_support type: enhancement * TriBITSPub/TriBITS#408: Deal with spaces in CDash url parts (SESW-383) component: ci_support type: enhancement * TriBITSPub/TriBITS#403: Spelling fixes * TriBITSPub/TriBITS#407: Move tribits_get_build_url_and_write_to_file() to stand-alone module (TriBITSPub/TriBITS#154) component: ctest_driver type: enhancement * TriBITSPub/TriBITS#388: Fixing typos (TriBITSPub/TriBITS#377) * TriBITSPub/TriBITS#406: Fix tribits_ctest_driver() package-by-package mode for CMake 3.19+ (TriBITSPub/TriBITS#363, TriBITSPub/TriBITS#394) component: ctest_driver type: bug * TriBITSPub/TriBITS#401: Improve GitHub Actions and CDash integration (TriBITSPub/TriBITS#154) type: enhancement * TriBITSPub/TriBITS#366: CI: draft action yaml * TriBITSPub/TriBITS#402: Revert some incorrect uppercase->lowercase changes * TriBITSPub/TriBITS#387: Build and deploy TriBITS documentation with Sphinx (TriBITSPub/TriBITS#386) component: documentation type: enhancement * TriBITSPub/TriBITS#398: Deal with pr null in not preprending build name (TriBITSPub/TriBITS#363) type: bug * TriBITSPub/TriBITS#396: Send PR results to 'Pull Request' CDash group and add PR ID to build names (TriBITSPub/TriBITS#363) type: enhancement * TriBITSPub/TriBITS#397: Print the ninja path and version (TriBITSPub/TriBITS#363) * TriBITSPub/TriBITS#393: GitHub Actions based testing for TriBITS (TriBITSPub/TriBITS#363) type: enhancement * TriBITSPub/TriBITS#389: TriBITS CI testing with GitHub Actions (TriBITSPub/TriBITS#363) type: enhancement * TriBITSPub/TriBITS#392: Fix broken tests for non-Fortran and CMake 3.21 builds (TriBITSPub/TriBITS#363) component: core * TriBITSPub/TriBITS#391: Fix up build_docs.sh for Sphinx doc generation (TriBITSPub/TriBITS#386) component: documentation type: enhancement * TriBITSPub/TriBITS#390: Add test for doc generation and fix usage of Python3 component: documentation type: bug * TriBITSPub/TriBITS#385: Replace last few references to TribitsDevelopersGuide.html (TriBITSPub/TriBITS#381) component: documentation type: enhancement * TriBITSPub/TriBITS#384: Split TribitsDevelopersGuide.* into TribitsUsersGuide.* and TribitsMaintainersGuide.* (TriBITSPub/TriBITS#381) component: documentation type: enhancement * TriBITSPub/TriBITS#383: Remove endfunction(<string>) and endmacro(<string>) (TriBITSPub/TriBITS#274, TriBITSPub/TriBITS#382) component: common_tpls type: bug * TriBITSPub/TriBITS#380: More package-arch data-structure documentation updates (TriBITSPub/TriBITS#63) component: documentation type: enhancement * TriBITSPub/TriBITS#379: Convert TriBITS to lower-case CMake command, macro, and function names (TriBITSPub/TriBITS#274) The following files were conflicting where I went with what is on the Trilinos 'develop' branch: * cmake/tribits/common_tpls/FindTPLBLAS.cmake * cmake/tribits/common_tpls/FindTPLLAPACK.cmake * cmake/tribits/common_tpls/FindTPLNetcdf.cmake (It looks like the above changes never made it back into TriBITS proper. The conflicts were due to the case changes in cmake command calls in these files due to TriBITSPub/TriBITS#379.) There was also a conflict in the file: * cmake/tribits/core/installation/TribitsProjectConfigTemplate.cmake.in I looked at that one carefully and I think that may have been due to fixes on both sides and then the case changes from TriBITSPub/TriBITS#379.

bartlettroscoe added the ready label Dec 5, 2016

bartlettroscoe mentioned this issue Jun 26, 2017

Add TriBITS support for all-at-once configure, build, test, and submit to CDash #183

Closed

bartlettroscoe mentioned this issue Aug 2, 2017

Implement automated pull request testing for Trilinos trilinos/Trilinos#1155

Closed

bartlettroscoe mentioned this issue May 1, 2018

Provide tools and approaches for robust determination of pass/fail of CTest -S driver scripts #261

Open

bartlettroscoe added a commit that referenced this issue Aug 8, 2018

Remove wrong CDash URL location

6db536a

This is not the right location. Need to implement #154.

bartlettroscoe added type: enhancement component: ctest_driver labels Jan 19, 2019

bartlettroscoe removed the stage: ready label Mar 15, 2019

bartlettroscoe self-assigned this Mar 28, 2019

bartlettroscoe changed the title ~~Improve TribitsCTestDriverCore.cmake for integration with GitHub Travis CI (and other such drivers)~~ Improve TribitsCTestDriverCore.cmake for integration with GitHub CI drivers Jul 21, 2021

bartlettroscoe mentioned this issue Jul 21, 2021

Set up TriBITS testing using GitHub Actions #363

Closed

16 tasks

bartlettroscoe added a commit to bartlettroscoe/TriBITS that referenced this issue Jul 27, 2021

WIP: Set CTEST_CHANGE_ID (TriBITSPub#154)

9345d53

bartlettroscoe added a commit to bartlettroscoe/TriBITS that referenced this issue Jul 27, 2021

Set CTEST_CHANGE_ID, CTEST_UPDATE_VERSION_ONLY, and CTEST_DO_UPDATES=…

204a609

…ON (TriBITSPub#154)

bartlettroscoe mentioned this issue Jul 27, 2021

Improve GitHub Actions and CDash integration (#154) #401

Merged

bartlettroscoe added a commit to bartlettroscoe/TriBITS that referenced this issue Jul 28, 2021

Update tribits_read_ctest_tag_file() to also read model (TriBITSPub#154)

5da54c8

Version of CMake 3.17 (and likely earlier) added the model in addition to group to the TAG file.

bartlettroscoe added a commit to bartlettroscoe/TriBITS that referenced this issue Jul 29, 2021

Update tribits_read_ctest_tag_file() to also read model (TriBITSPub#154)

1a9efd1

Version of CMake 3.17 (and likely earlier) added the model in addition to group to the TAG file.

bartlettroscoe added a commit to bartlettroscoe/TriBITS that referenced this issue Jul 29, 2021

WIP: Change model and CDash group for URL file creation (TriBITSPub#154)

dba542b

I will squash this comit with commits that complete the system-level test for a build-specific URL (TriBITSPub#154).

bartlettroscoe added a commit to bartlettroscoe/TriBITS that referenced this issue Jul 29, 2021

WIP: Add skeleton and failing test for building CDash build URL (TriB…

04c2390

…ITSPub#154) I will amend/squash this commit with the full implementation.

bartlettroscoe added a commit to bartlettroscoe/TriBITS that referenced this issue Aug 9, 2021

Implement tribits_get_cdash_build_url_from_parts() (TriBITSPub#154)

d7f2306

bartlettroscoe added a commit to bartlettroscoe/TriBITS that referenced this issue Aug 10, 2021

Implement tribits_get_cdash_build_url_from_tag_file() (TriBITSPub#154)

f3ffa30

bartlettroscoe added a commit to bartlettroscoe/TriBITS that referenced this issue Aug 10, 2021

Implement tribits_get_cdash_index_php_from_drop_site_and_location() (T…

b817c75

…riBITSPub#154) Need this for calling tribits_get_cdash_build_url_from_tag_file() from tribits_ctest_driver().

bartlettroscoe added a commit to bartlettroscoe/TriBITS that referenced this issue Aug 10, 2021

Construct and print CDash build URL and write file CDashBuildUrl.txt (T…

50d6e3a

…riBITSPub#154)

bartlettroscoe added a commit to bartlettroscoe/TriBITS that referenced this issue Aug 10, 2021

Add GHA step to print URL to results on CDash (TriBITSPub#154)

a6d7e71

bartlettroscoe added a commit to bartlettroscoe/TriBITS that referenced this issue Aug 10, 2021

Update tribits_read_ctest_tag_file() to also read model (TriBITSPub#154)

0ce719a

Version of CMake 3.17 (and likely earlier) added the model in addition to group to the TAG file.

bartlettroscoe added a commit to bartlettroscoe/TriBITS that referenced this issue Aug 10, 2021

Implement tribits_get_cdash_build_url_from_parts() (TriBITSPub#154)

79a5064

bartlettroscoe added a commit to bartlettroscoe/TriBITS that referenced this issue Aug 10, 2021

Implement tribits_get_cdash_build_url_from_tag_file() (TriBITSPub#154)

0f983ef

bartlettroscoe added a commit to bartlettroscoe/TriBITS that referenced this issue Aug 10, 2021

Implement tribits_get_cdash_index_php_from_drop_site_and_location() (T…

687c1a3

…riBITSPub#154) Need this for calling tribits_get_cdash_build_url_from_tag_file() from tribits_ctest_driver().

bartlettroscoe added a commit to bartlettroscoe/TriBITS that referenced this issue Aug 10, 2021

Construct and print CDash build URL and write file CDashBuildUrl.txt (T…

23189b9

…riBITSPub#154)

bartlettroscoe added a commit to bartlettroscoe/TriBITS that referenced this issue Aug 10, 2021

Add GHA step to print URL to results on CDash (TriBITSPub#154)

60bf144

bartlettroscoe closed this as completed Aug 12, 2021

bartlettroscoe mentioned this issue Aug 13, 2021

Move tribits_get_build_url_and_write_to_file() to stand-alone module (#154) #407

Merged

bartlettroscoe added a commit that referenced this issue Aug 13, 2021

Merge pull request #407 from bartlettroscoe/154-stand-alone-cdash-url…

576f455

…-module.cmake Move tribits_get_build_url_and_write_to_file() to stand-alone module (#154)

bartlettroscoe mentioned this issue Sep 4, 2021

TriBITS Snapshot update as of 2021-09-03 trilinos/Trilinos#9663

Merged

bartlettroscoe added this to TriBITS Refactor Jun 18, 2024

bartlettroscoe moved this to Done in TriBITS Refactor Jun 18, 2024

bartlettroscoe added this to TriBITS Jun 18, 2024

bartlettroscoe moved this to Done in TriBITS Jun 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve TribitsCTestDriverCore.cmake for integration with GitHub CI drivers #154

Improve TribitsCTestDriverCore.cmake for integration with GitHub CI drivers #154

bartlettroscoe commented Nov 10, 2016 •

edited

Loading

bartlettroscoe commented Sep 18, 2017

bartlettroscoe commented May 5, 2020 •

edited

Loading

bartlettroscoe commented Aug 11, 2021

bartlettroscoe commented Aug 11, 2021

bartlettroscoe commented Aug 12, 2021

bartlettroscoe commented Aug 16, 2021

Improve TribitsCTestDriverCore.cmake for integration with GitHub CI drivers #154

Improve TribitsCTestDriverCore.cmake for integration with GitHub CI drivers #154

Comments

bartlettroscoe commented Nov 10, 2016 • edited Loading

Description

bartlettroscoe commented Sep 18, 2017

bartlettroscoe commented May 5, 2020 • edited Loading

bartlettroscoe commented Aug 11, 2021

bartlettroscoe commented Aug 11, 2021

bartlettroscoe commented Aug 12, 2021

bartlettroscoe commented Aug 16, 2021

bartlettroscoe commented Nov 10, 2016 •

edited

Loading

bartlettroscoe commented May 5, 2020 •

edited

Loading