Skip to content

run package uploads in python slim containers #586

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rapids-bot merged 5 commits into from
Jun 29, 2026
Merged

run package uploads in python slim containers #586

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
2f33ba1
run package uploads in python slim containers
jameslamb Jun 29, 2026
8b390aa
simplify tool installs, fix cloning
jameslamb Jun 29, 2026
1aca284
Update .github/workflows/conda-upload-packages.yaml
jameslamb Jun 29, 2026
0acff9b
use 'main' of gha-tools
jameslamb Jun 29, 2026
c395258
oh Github browser editing, why are you so weird about whitespace
jameslamb Jun 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 37 additions & 6 deletions .github/workflows/conda-upload-packages.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,11 +67,46 @@ jobs:
upload:
runs-on: linux-amd64-cpu4
container:
image: rapidsai/ci-conda:26.08-latest # zizmor: ignore[unpinned-images]
image: "python:3.14-slim"
env:
RAPIDS_BUILD_TYPE: ${{ inputs.build_type }}
steps:

- name: Setup proxy cache
uses: nv-gha-runners/setup-proxy-cache@main
continue-on-error: true
with:
enable-apt: true

- name: Install tools
run: |
# install tools needed to bootstrap others
apt-get update
apt-get install -y --no-install-recommends \
curl \
git

# install gha-tools
git clone --depth 1 https://github.com/rapidsai/gha-tools /tmp/gha-tools
echo "/tmp/gha-tools/tools" >> "${GITHUB_PATH}"
export PATH="/tmp/gha-tools/tools:${PATH}"

# set up source for latest 'gh' releases
curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg
chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | tee /etc/apt/sources.list.d/github-cli.list

# install tools with system package manager
apt-get update
apt-get install -y --no-install-recommends \
gh \
jq

# anaconda client (for 'rapids-upload-to-anaconda-github')
rapids-pip-retry install \
--prefer-binary \
'anaconda-client>=1.13.1'

- name: Standardize repository information
uses: rapidsai/shared-actions/rapids-github-info@main
with:
Expand All @@ -87,11 +122,7 @@ jobs:
# checking '/rate_limit | jq .' should not itself count against any rate limits.
- name: Check GitHub API rate limits
run: |
if ! type gh >/dev/null; then
echo "'gh' CLI is not installed... skipping rate-limits check"
else
gh api /rate_limit | jq .
fi
gh api /rate_limit | jq .
env:
GH_TOKEN: ${{ github.token }}
- name: Set Proper Conda Upload Token
Expand Down
45 changes: 38 additions & 7 deletions .github/workflows/wheels-publish.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,11 +79,46 @@ jobs:
container:
# CUDA toolkit version of the container is irrelevant in the publish step.
# This just uploads already-built wheels to remote storage.
image: "rapidsai/ci-wheel:26.08-latest" # zizmor: ignore[unpinned-images]
image: "python:3.14-slim"

@jameslamb jameslamb Jun 29, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't meaningfully be impacted by Python version here, as long as it's an officially-supported version.

I chose not to use python:latest here to reduce the risk of these builds failing if :latest becomes (for example) 3.16.0rc1 and there aren't yet compatible wheels from some dependencies we pip install.

env:
RAPIDS_BUILD_TYPE: ${{ inputs.build_type }}
steps:

- name: Setup proxy cache
uses: nv-gha-runners/setup-proxy-cache@main
continue-on-error: true
with:
enable-apt: true

@jameslamb jameslamb Jun 29, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that code in this workflow runs apt-get install and pip install, we'll want the proxy cache for faster downloads and reduced risk of network issues.


- name: Install tools
run: |
# install tools needed to bootstrap others
apt-get update
apt-get install -y --no-install-recommends \
curl \
git

# install gha-tools
git clone --depth 1 https://github.com/rapidsai/gha-tools /tmp/gha-tools
echo "/tmp/gha-tools/tools" >> "${GITHUB_PATH}"
export PATH="/tmp/gha-tools/tools:${PATH}"

@jameslamb jameslamb Jun 29, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GITHUB_PATH doesn't take effect until the next step, but we want gha-tools to be on PATH for rapids-pip-retry in this step.


# set up source for latest 'gh' releases
curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg
chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | tee /etc/apt/sources.list.d/github-cli.list

# install tools with system package manager
apt-get update
apt-get install -y --no-install-recommends \
gh \
jq

# anaconda client (for 'rapids-upload-to-anaconda-github')
rapids-pip-retry install \
--prefer-binary \
'anaconda-client>=1.13.1'

- name: Standardize repository information
uses: rapidsai/shared-actions/rapids-github-info@main
with:
Expand All @@ -99,11 +134,7 @@ jobs:
# checking '/rate_limit | jq .' should not itself count against any rate limits.
- name: Check GitHub API rate limits
run: |
if ! type gh >/dev/null; then
echo "'gh' CLI is not installed... skipping rate-limits check"
else
gh api /rate_limit | jq .
fi
gh api /rate_limit | jq .

@jameslamb jameslamb Jun 29, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're now explicitly installing gh as part of this workflow. gh not being available would be a bug and we should get a loud error telling us about it.

env:
GH_TOKEN: ${{ github.token }}

Expand Down Expand Up @@ -140,7 +171,7 @@ jobs:
- name: Publish the downloaded wheels to PyPI
if: ${{ inputs.publish_to_pypi && steps.check_if_release.outputs.is_release_build == 'true' }}
run: |
python3 -m pip install twine
python3 -m pip install 'twine>=6.2.0'
python3 -m twine upload -u __token__ "${UPLOAD_DIR}"/*
env:
UPLOAD_DIR: ${{ steps.download_wheels.outputs.package_upload_dir }}
Expand Down