[NEVER MERGE] remove --skip-existing from upload tools - #265
Closed
jameslamb wants to merge 2 commits into
Closed
Conversation
rapids-bot Bot
pushed a commit
to rapidsai/shared-workflows
that referenced
this pull request
Jun 29, 2026
Contributes to #505 As described there, it's not uncommon for pulling the `rapidsai/ci-wheel` images to take 3-4 minutes. This is extra painful for jobs like `wheels-publish`, which just need lightweight publishing tools and not CUDA libraries, compiler toolchain, etc. This PR proposes working around that by doing the following: * using a small `python:3.14-slim` image instead * installing just the small set of necessary tools at runtime of the `wheels-publish` job ## Notes for Reviewers ### Benefits * faster `conda-uploads-packages` and `wheels-publish`, which means reduced end-to-end time for RAPIDS nightly pipeline (see "how I tested this") * would allow us to stop installing `anaconda-client` and its dependencies in the `rapidsai/ci-conda` and `rapidsai/ci-wheel` images at https://github.com/rapidsai/ci-imgs - _that project has a lot of dependencies, so that might have a small but notable improvement in image size, build time, and pull time_ - _that'd affect all wheel builds and conda building + testing jobs_ ### Costs / Risks * adds more network calls and package installs at runtime of `wheels-publish`, which might lead to more transient failures from network issues * adds complexity to the workflow code ### Why not pre-build a `wheels-publish` image? That WOULD make this even faster and avoid all those package installs at runtime. But new images need to go through a compliance/legal approval process that's a bit heavier than this use case justifies, in my opinion. In my testing, installing tools took 15-20 seconds, so that's the most we'd save by having a pre-built image with everything installed. ### How I tested this `rmm`'s `main` branch was already pointed at this branch for package-uploading jobs from #585. Put up changes in a `gha-tools` branch (rapidsai/gha-tools#265) to force-overwrite existing packages, so the upload time to anaconda.org is included in the timings. Clicked "re-run all jobs" on `rmm`'s most recent run on `main`. | workflow | previous `main` (as of #585) | this PR | |----------------------------------:|:-----------------------------------------:|:------------------:| | upload-conda | 1m32s | **1m15s** | | wheels-publish-cpp | 2m51s | **0m37s** | | wheels-publish-python | 3m3s | **0m41s** | *builds from `rmm`: ([previous main](https://github.com/rapidsai/rmm/actions/runs/28388699537/attempts/2) | [this PR](https://github.com/rapidsai/rmm/actions/runs/28388699537/job/84139196301))* Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #586
Member
Author
|
This testing is done, this PR can be closed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Never merge this... just opening to show the changes being tested in rapidsai/shared-workflows#586