Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch 3rdparty directory to only download files if YCM_DOWNLOAD_3RDPARTY is set to ON #402

Merged
merged 3 commits into from
Apr 7, 2022

Conversation

traversaro
Copy link
Member

If YCM_DOWNLOAD_3RDPARTY is set to OFF, the already downloaded files committed in the repo are used instead.

The main goal of this is to shift the burden of downloading new 3rdparty files from the users to the mantainer, that once in a while will need to update the files and run the build with YCM_DOWNLOAD_3RDPARTY option enabled. This should reduce the time necessary to install YCM from source, including when YCM is used for bootstrap. Furtermore, it reduces the possibility that the YCM build fails, as even if gitlab.org , gitlab.kitware.com or invent.kde.org are down, the YCM build will successed anyhow.

This should also have the side effect of speeding up the build of YCM.

Fix #105 .

@traversaro
Copy link
Member Author

@PeterBowman any opinion on this? Asking as you commented in the related issue in #105 (comment) .

@PeterBowman
Copy link
Member

@PeterBowman any opinion on this? Asking as you commented in the related issue in #105 (comment) .

I think I've not seen that error (GitLab was down and UseLatex.cmake could not be downloaded) in a long time, but this is great as it shields us against similar issues in the future.

@traversaro
Copy link
Member Author

traversaro commented Jul 11, 2022

Example of problem avoided after this PR (from https://github.com/robotology/gazebo-yarp-plugins/runs/7275245883):

2022-07-11T02:46:18.8271856Z [ 86%] Downloading file find-modules/FindUDev.cmake from KDE Extra CMake Modules git repository (ref v5.79.0)
2022-07-11T02:46:19.4341128Z -- Cannot download file https://invent.kde.org/frameworks/extra-cmake-modules/-/raw/v5.79.0/find-modules/FindUDev.cmake
2022-07-11T02:46:19.4342252Z   Network problem or not existing file.
2022-07-11T02:46:19.4343264Z   CMake Error at /home/runner/work/gazebo-yarp-plugins/gazebo-yarp-plugins/ycm/build/3rdparty/CMakeFiles/3rdparty-ecm.dir/ycm_download_find_modules_FindUDev_cmake_real.cmake:9 (file):
2022-07-11T02:46:19.4343808Z   file DOWNLOAD HASH mismatch
2022-07-11T02:46:19.4344002Z 
2022-07-11T02:46:19.4344556Z     for file: [/home/runner/work/gazebo-yarp-plugins/gazebo-yarp-plugins/ycm/build/3rdparty/CMakeFiles/3rdparty-ecm.dir/downloads/find-modules/FindUDev.cmake]
2022-07-11T02:46:19.4345108Z       expected hash: [10bfe244e8f2c4d85338563a0057656093fb0d08]
2022-07-11T02:46:19.4345498Z         actual hash: [da39a3ee5e6b4b0d3255bfef95601890afd80709]
2022-07-11T02:46:19.4345874Z              status: [22;"HTTP response code said error"].
2022-07-11T02:46:19.4346148Z   Retrying.
2022-07-11T02:46:19.4346299Z 
2022-07-11T02:46:19.8081774Z -- Cannot download file https://invent.kde.org/frameworks/extra-cmake-modules/-/raw/v5.79.0/find-modules/FindUDev.cmake
2022-07-11T02:46:19.8086636Z   Network problem or not existing file.
2022-07-11T02:46:19.8087364Z   CMake Error at /home/runner/work/gazebo-yarp-plugins/gazebo-yarp-plugins/ycm/build/3rdparty/CMakeFiles/3rdparty-ecm.dir/ycm_download_find_modules_FindUDev_cmake_real.cmake:9 (file):
2022-07-11T02:46:19.8087815Z   file DOWNLOAD HASH mismatch
2022-07-11T02:46:19.8087976Z 
2022-07-11T02:46:19.8088453Z     for file: [/home/runner/work/gazebo-yarp-plugins/gazebo-yarp-plugins/ycm/build/3rdparty/CMakeFiles/3rdparty-ecm.dir/downloads/find-modules/FindUDev.cmake]
2022-07-11T02:46:19.8088936Z       expected hash: [10bfe244e8f2c4d85338563a0057656093fb0d08]
2022-07-11T02:46:19.8089265Z         actual hash: [da39a3ee5e6b4b0d3255bfef95601890afd80709]
2022-07-11T02:46:19.8089600Z              status: [22;"HTTP response code said error"].
2022-07-11T02:46:19.8089854Z   Retrying.
2022-07-11T02:46:19.8089987Z 
2022-07-11T02:46:20.1885901Z CMake Error at /home/runner/work/gazebo-yarp-plugins/gazebo-yarp-plugins/ycm/build/3rdparty/CMakeFiles/3rdparty-ecm.dir/ycm_download_find_modules_FindUDev_cmake.cmake:30 (message):
2022-07-11T02:46:20.1886779Z   Cannot download file
2022-07-11T02:46:20.1887684Z   https://invent.kde.org/frameworks/extra-cmake-modules/-/raw/v5.79.0/find-modules/FindUDev.cmake
2022-07-11T02:46:20.1888208Z 
2022-07-11T02:46:20.1888364Z 
2022-07-11T02:46:20.1888658Z     Network problem or not existing file.
2022-07-11T02:46:20.1889633Z     CMake Error at /home/runner/work/gazebo-yarp-plugins/gazebo-yarp-plugins/ycm/build/3rdparty/CMakeFiles/3rdparty-ecm.dir/ycm_download_find_modules_FindUDev_cmake_real.cmake:9 (file):
2022-07-11T02:46:20.1890281Z     file DOWNLOAD HASH mismatch
2022-07-11T02:46:20.1890532Z 
2022-07-11T02:46:20.1892767Z 3rdparty/CMakeFiles/3rdparty-ecm.dir/build.make:102: recipe for target '3rdparty/ecm/find-modules/FindUDev.cmake' failed
2022-07-11T02:46:20.1895017Z CMakeFiles/Makefile2:1397: recipe for target '3rdparty/CMakeFiles/3rdparty-ecm.dir/all' failed
2022-07-11T02:46:20.1896501Z   
2022-07-11T02:46:20.1896902Z 
2022-07-11T02:46:20.1949730Z       for file: [/home/runner/work/gazebo-yarp-plugins/gazebo-yarp-plugins/ycm/build/3rdparty/CMakeFiles/3rdparty-ecm.dir/downloads/find-modules/FindUDev.cmake]
2022-07-11T02:46:20.1950230Z         expected hash: [10bfe244e8f2c4d85338563a0057656093fb0d08]
2022-07-11T02:46:20.1950560Z           actual hash: [da39a3ee5e6b4b0d3255bfef95601890afd80709]
2022-07-11T02:46:20.1950895Z                status: [22;"HTTP response code said error"]
2022-07-11T02:46:20.1951076Z 
2022-07-11T02:46:20.1951092Z 
2022-07-11T02:46:20.1951343Z make[2]: *** [3rdparty/ecm/find-modules/FindUDev.cmake] Error 1
2022-07-11T02:46:20.1951767Z make[1]: *** [3rdparty/CMakeFiles/3rdparty-ecm.dir/all] Error 2
GitHub
GitHub is where people build software. More than 83 million people use GitHub to discover, fork, and contribute to over 200 million projects.

@traversaro
Copy link
Member Author

Example of problem avoided after this PR (from https://github.com/robotology/gazebo-yarp-plugins/runs/7275245883):

2022-07-11T02:46:18.8271856Z [ 86%] Downloading file find-modules/FindUDev.cmake from KDE Extra CMake Modules git repository (ref v5.79.0)
2022-07-11T02:46:19.4341128Z -- Cannot download file https://invent.kde.org/frameworks/extra-cmake-modules/-/raw/v5.79.0/find-modules/FindUDev.cmake
2022-07-11T02:46:19.4342252Z   Network problem or not existing file.
2022-07-11T02:46:19.4343264Z   CMake Error at /home/runner/work/gazebo-yarp-plugins/gazebo-yarp-plugins/ycm/build/3rdparty/CMakeFiles/3rdparty-ecm.dir/ycm_download_find_modules_FindUDev_cmake_real.cmake:9 (file):
2022-07-11T02:46:19.4343808Z   file DOWNLOAD HASH mismatch
2022-07-11T02:46:19.4344002Z 
2022-07-11T02:46:19.4344556Z     for file: [/home/runner/work/gazebo-yarp-plugins/gazebo-yarp-plugins/ycm/build/3rdparty/CMakeFiles/3rdparty-ecm.dir/downloads/find-modules/FindUDev.cmake]
2022-07-11T02:46:19.4345108Z       expected hash: [10bfe244e8f2c4d85338563a0057656093fb0d08]
2022-07-11T02:46:19.4345498Z         actual hash: [da39a3ee5e6b4b0d3255bfef95601890afd80709]
2022-07-11T02:46:19.4345874Z              status: [22;"HTTP response code said error"].
2022-07-11T02:46:19.4346148Z   Retrying.
2022-07-11T02:46:19.4346299Z 
2022-07-11T02:46:19.8081774Z -- Cannot download file https://invent.kde.org/frameworks/extra-cmake-modules/-/raw/v5.79.0/find-modules/FindUDev.cmake
2022-07-11T02:46:19.8086636Z   Network problem or not existing file.
2022-07-11T02:46:19.8087364Z   CMake Error at /home/runner/work/gazebo-yarp-plugins/gazebo-yarp-plugins/ycm/build/3rdparty/CMakeFiles/3rdparty-ecm.dir/ycm_download_find_modules_FindUDev_cmake_real.cmake:9 (file):
2022-07-11T02:46:19.8087815Z   file DOWNLOAD HASH mismatch
2022-07-11T02:46:19.8087976Z 
2022-07-11T02:46:19.8088453Z     for file: [/home/runner/work/gazebo-yarp-plugins/gazebo-yarp-plugins/ycm/build/3rdparty/CMakeFiles/3rdparty-ecm.dir/downloads/find-modules/FindUDev.cmake]
2022-07-11T02:46:19.8088936Z       expected hash: [10bfe244e8f2c4d85338563a0057656093fb0d08]
2022-07-11T02:46:19.8089265Z         actual hash: [da39a3ee5e6b4b0d3255bfef95601890afd80709]
2022-07-11T02:46:19.8089600Z              status: [22;"HTTP response code said error"].
2022-07-11T02:46:19.8089854Z   Retrying.
2022-07-11T02:46:19.8089987Z 
2022-07-11T02:46:20.1885901Z CMake Error at /home/runner/work/gazebo-yarp-plugins/gazebo-yarp-plugins/ycm/build/3rdparty/CMakeFiles/3rdparty-ecm.dir/ycm_download_find_modules_FindUDev_cmake.cmake:30 (message):
2022-07-11T02:46:20.1886779Z   Cannot download file
2022-07-11T02:46:20.1887684Z   https://invent.kde.org/frameworks/extra-cmake-modules/-/raw/v5.79.0/find-modules/FindUDev.cmake
2022-07-11T02:46:20.1888208Z 
2022-07-11T02:46:20.1888364Z 
2022-07-11T02:46:20.1888658Z     Network problem or not existing file.
2022-07-11T02:46:20.1889633Z     CMake Error at /home/runner/work/gazebo-yarp-plugins/gazebo-yarp-plugins/ycm/build/3rdparty/CMakeFiles/3rdparty-ecm.dir/ycm_download_find_modules_FindUDev_cmake_real.cmake:9 (file):
2022-07-11T02:46:20.1890281Z     file DOWNLOAD HASH mismatch
2022-07-11T02:46:20.1890532Z 
2022-07-11T02:46:20.1892767Z 3rdparty/CMakeFiles/3rdparty-ecm.dir/build.make:102: recipe for target '3rdparty/ecm/find-modules/FindUDev.cmake' failed
2022-07-11T02:46:20.1895017Z CMakeFiles/Makefile2:1397: recipe for target '3rdparty/CMakeFiles/3rdparty-ecm.dir/all' failed
2022-07-11T02:46:20.1896501Z   
2022-07-11T02:46:20.1896902Z 
2022-07-11T02:46:20.1949730Z       for file: [/home/runner/work/gazebo-yarp-plugins/gazebo-yarp-plugins/ycm/build/3rdparty/CMakeFiles/3rdparty-ecm.dir/downloads/find-modules/FindUDev.cmake]
2022-07-11T02:46:20.1950230Z         expected hash: [10bfe244e8f2c4d85338563a0057656093fb0d08]
2022-07-11T02:46:20.1950560Z           actual hash: [da39a3ee5e6b4b0d3255bfef95601890afd80709]
2022-07-11T02:46:20.1950895Z                status: [22;"HTTP response code said error"]
2022-07-11T02:46:20.1951076Z 
2022-07-11T02:46:20.1951092Z 
2022-07-11T02:46:20.1951343Z make[2]: *** [3rdparty/ecm/find-modules/FindUDev.cmake] Error 1
2022-07-11T02:46:20.1951767Z make[1]: *** [3rdparty/CMakeFiles/3rdparty-ecm.dir/all] Error 2

GitHub**Build software better, together**GitHub is where people build software. More than 83 million people use GitHub to discover, fork, and contribute to over 200 million projects.

In this specific case, https://invent.kde.org/frameworks/extra-cmake-modules/-/raw/v5.79.0/find-modules/FindUDev.cmake is marked as 403, but all other files are accessible (see for example https://invent.kde.org/frameworks/extra-cmake-modules/-/raw/v5.80.0/find-modules/FindUDev.cmake). I wonder if YCM (and its use in CI systems) played a role in the reason why that URL is now not accessible. I guess that anyhow we should not risk these problems anymore after this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bootstrap with offline version?
4 participants