Download build artifacts from Github for CI jobs - #18539
Conversation
jayavenkatesh19
added
2 - In Progress
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
/ok to test |
jayavenkatesh19
changed the title
jayavenkatesh19
added
3 - Ready for Review
jameslamb
left a comment
jameslamb
left a comment
There was a problem hiding this comment.
Great! I looked through build and test logs for both conda and wheels and saw what I expected... the downloaded built-in-CI artifacts, not nightlies, were getting installed everywhere.
I left one question that I'd like answered before I approve this.
| LIBCUDF_WHEELHOUSE=$(RAPIDS_PY_WHEEL_NAME="libcudf_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-github cpp) | ||
| PYLIBCUDF_WHEELHOUSE=$(RAPIDS_PY_WHEEL_NAME="pylibcudf_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-github python) | ||
| echo "libcudf-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo ${LIBCUDF_WHEELHOUSE}/libcudf_*.whl)" > /tmp/constraints.txt | ||
| echo "pylibcudf-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo ${PYLIBCUDF_WHEELHOUSE}/pylibcudf_*.whl)" >> /tmp/constraints.txt |
There was a problem hiding this comment.
Why is this PR only changing the rapids-download-* portion of CI scripts, and not also removing the upload-to-S3 logic? I'd expected all of the S3 uploading and downloading to get removed at the same time, once we were confident that GitHub artifact store is working for us.
For example, if nothing in CI calls rapids-download-wheels-from-s3 any more, what is the value of keeping lines like this?
There was a problem hiding this comment.
This was discussed here as part of the roll out plan: https://github.com/rapidsai/ops/issues/2982#issuecomment-2669867781
The main reason to do so was that developers might some tooling dependent on downloading build artifacts from S3, and now that everyone has been notified that we are switching over to Github Artifacts, this gives everyone some time to adopt the newer paradigm, before we finally stop uploading to S3 altogether.
There was a problem hiding this comment.
Thanks. I did see that comment. It says
- Migrate each repository to download artifacts from GitHub Artifacts (one PR per repo)
- Stop publishing build artifacts to downloads.rapids.ai
Nothing in that comment suggested to me that the plan was for each item in that list to be a separate round of touch-every-RAPIDS-repo PRs. Those are expensive... they take up CI resources and reviewing time. I would have removed uploading to and downloading from S3 in the same round of PRs.
BUT... all that said, if this is the plan you, @ajschmidt8 , and @bdice have already agreed to, and since the developer-facing docs updates showing people how to migrate their local workflows aren't yet done (rapidsai/docs#593), I'll approve this and the other PRs like it and get out of the way.
|
/merge |
3 participants
Description
This work is towards moving build artifacts from
downloads.rapids.aito Github Artifact Store (see https://github.com/rapidsai/build-infra/issues/237)Updates conda and wheel artifact download source from S3 to GitHub across CI scripts.
Uses dynamic temporary paths for wheel downloads returned by
rapids-download-wheels-from-githubinstead of using fixed directories, to streamline wheel downloads to conda downloads.Also updates CI workflows to follow
package-nameconvention between wheel build and wheel publish jobs.Checklist