Skip to content

add libcuml Python builds#442

Merged
vyasr merged 3 commits intorapidsai:branch-25.02from
jameslamb:libcuml-wheels
Jan 23, 2025
Merged

add libcuml Python builds#442
vyasr merged 3 commits intorapidsai:branch-25.02from
jameslamb:libcuml-wheels

Conversation

@jameslamb
Copy link
Member

@jameslamb jameslamb commented Jan 22, 2025 •
edited
Loading

Contributes to rapidsai/build-planning#33

Adjusts rapids-build-utils manifest for release 25.02 to account for the introduction of new libcuml wheels (rapidsai/cuml#6199).

Notes for Reviewers

This shouldn't be merged still pointing at my forks.

Plan:

  1. merge add libcuvs Python builds #440, then update this branch
  2. see CI pass here
  3. see all CI except devcontainers pass on introduce libcuml wheels cuml#6199
  4. point this PR back at upstream and admin-merge it
  5. re-run devcontainers CI on introduce libcuml wheels cuml#6199 and see it pass

@jameslamb jameslamb added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Jan 22, 2025
@jameslamb jameslamb mentioned this pull request Jan 22, 2025
Merged
bdice
bdice approved these changes Jan 23, 2025
View reviewed changes
Copy link
Contributor

@bdice bdice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good. Let me know when you want to revert the testing changes and merge this.

@jameslamb
Copy link
Member Author

jameslamb commented Jan 23, 2025

🎉 we have a successful build (build link)

I think that's good evidence that the changes from rapidsai/cuml#6199 won't break devcontainers. Going to revert my point-at-my-fork changes here and mark this ready for review.

@jameslamb jameslamb changed the title WIP: add libcuml Python builds add libcuml Python builds Jan 23, 2025
@jameslamb jameslamb marked this pull request as ready for review January 23, 2025 17:58
@jameslamb jameslamb requested a review from a team as a code owner January 23, 2025 17:58
@jameslamb jameslamb requested review from AyodeAwe and removed request for a team January 23, 2025 17:58
@jameslamb jameslamb removed the request for review from AyodeAwe January 23, 2025 21:17
@vyasr vyasr merged commit 38d7e39 into rapidsai:branch-25.02 Jan 23, 2025
22 of 24 checks passed
@jameslamb jameslamb deleted the libcuml-wheels branch January 23, 2025 22:26
rapids-bot bot pushed a commit to rapidsai/cuml that referenced this pull request Jan 24, 2025
@jameslamb
introduce libcuml wheels (#6199)
732dac9 
Replaces #6006, contributes to rapidsai/build-planning#33.

Proposes packaging `libcuml` as a wheel, which is then re-used by `cuml-cu{11,12}` wheels.

## Notes for Reviewers

### Benefits of these changes

* smaller wheels (see "Size Changes" below)
* faster compile times
  - *no more re-compiling RAFT, thanks to rapidsai/raft#2531
* less use of CI resources (only compiling once per CPU architecture / CUDA versions, instead of once per those + Python minor version)
* other benefits mentioned in rapidsai/build-planning#33

### Wheel contents

`libcuml`:

* `libcuml++.so` (shared library) and its headers
* `libcumlprims_mg.so` (shared library) and its headers
* other vendored dependencies (CCCL, `fmt`)

`cuml`:

* `cuml` Python / Cython code and compiled Cython extensions

### Dependency Flows

In short.... `libcuml` contains `libcuml.so` and `libcumlprims_mg.so` dynamic libraries and the headers to link against them.

* Anything that needs to link against cuML at build time pulls in `libcugraph` wheels as a build dependency.
* Anything that needs cuML's symbols at runtime pulls it in as a runtime dependency, and calls `libcuml.load_library()`.

For more details and some flowcharts, see rapidsai/build-planning#33 (comment)

### Size changes (CUDA 12, Python 3.12, x86_64)

| wheel                | num files (before) | num files (this PR) | size (before)  | size (this PR) |
|:---------------:|------------------:|-----------------:|--------------:|-------------:|
| `libcuml`           |   ---                       |   1766                   | ---                   | 289M                 |
| `cuml`               |   442                     |   441                    | 527M               | 9M                 |
|**TOTAL**          |   **442**              |   **2207**               | **527M**        | **298M**    |

*NOTES: size = compressed, "before" = 2025-01-22 nightlies*

<details><summary>how I calculated those (click me)</summary>

```shell
docker run \
    --rm \
    --network host \
    --env RAPIDS_NIGHTLY_DATE=2025-01-22 \
    --env CUML_NIGHTLY_SHA=01e19bba9821954b062a04fbf31d3522afa4b0b1 \
    --env CUML_PR="pull-request/6199" \
    --env CUML_PR_SHA="9d5100ec4589e20230a31817518427efa1e49c6d" \
    --env RAPIDS_PY_CUDA_SUFFIX=cu12 \
    --env WHEEL_DIR_BEFORE=/tmp/wheels-before \
    --env WHEEL_DIR_AFTER=/tmp/wheels-after \
    -it rapidsai/ci-wheel:cuda12.5.1-rockylinux8-py3.12 \
    bash

# --- nightly wheels --- #
mkdir -p ./wheels-before

export RAPIDS_BUILD_TYPE=branch
export RAPIDS_REF_NAME="branch-25.02"

# cuml
RAPIDS_PY_WHEEL_NAME="cuml_${RAPIDS_PY_CUDA_SUFFIX}" \
RAPIDS_REPOSITORY=rapidsai/cuml \
RAPIDS_SHA=${CUML_NIGHTLY_SHA} \
    rapids-download-wheels-from-s3 python ./wheels-before

# --- wheels from CI --- #
mkdir -p ./wheels-after

export RAPIDS_BUILD_TYPE="pull-request"

# libcuml
RAPIDS_PY_WHEEL_NAME="libcuml_${RAPIDS_PY_CUDA_SUFFIX}" \
RAPIDS_REPOSITORY=rapidsai/cuml \
RAPIDS_REF_NAME="${CUML_PR}" \
RAPIDS_SHA="${CUML_PR_SHA}" \
    rapids-download-wheels-from-s3 cpp ./wheels-after

# cuml
RAPIDS_PY_WHEEL_NAME="cuml_${RAPIDS_PY_CUDA_SUFFIX}" \
RAPIDS_REPOSITORY=rapidsai/cuml \
RAPIDS_REF_NAME="${CUML_PR}" \
RAPIDS_SHA="${CUML_PR_SHA}" \
    rapids-download-wheels-from-s3 python ./wheels-after

pip install pydistcheck
pydistcheck \
    --inspect \
    --select 'distro-too-large-compressed' \
    ./wheels-before/*.whl \
| grep -E '^checking|files: | compressed' \
> ./before.txt

# get more exact sizes
du -sh ./wheels-before/*

pydistcheck \
    --inspect \
    --select 'distro-too-large-compressed' \
    ./wheels-after/*.whl \
| grep -E '^checking|files: | compressed' \
> ./after.txt

# get more exact sizes
du -sh ./wheels-after/*
```

</details>

### How I tested this

These other PRs:

* rapidsai/devcontainers#442

Authors:
  - James Lamb (https://github.com/jameslamb)
  - Bradley Dice (https://github.com/bdice)

Approvers:
  - Bradley Dice (https://github.com/bdice)
  - Divye Gala (https://github.com/divyegala)

URL: #6199
@bdice bdice mentioned this pull request Apr 22, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bdice bdice bdice approved these changes

Assignees

No one assigned

Labels

improvement Improves an existing functionality non-breaking Introduces a non-breaking change

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jameslamb @bdice @vyasr