Migrate to use cuVS for vector search #6085

benfred · 2024-09-26T04:31:06Z

This PR updates to use cuVS instead of RAFT for vector search, pairwise distances and clustering. This is required for us to deprecate the vector search functionality in RAFT, in favour of the code in cuVS.

Because some code hasn't been migrated over to cuvs yet, we will continue to use the version in RAFT - but with RAFT in header only mode. In particular this functionality will be used in RAFT header only mode:

Random Ball Cover (see Moving random ball cover cuvs#218)
Sparse KNN
nn-descent [FEA] Migrate nn-descent updates from RAFT cuvs#364
MetricProcessor
knn_merge_parts
build_dendrogram_host
build_sorted_mst
raft DistanceType

Because sparse KNN in RAFT uses the DistanceType in RAFT, we can't fully move over to use the DistanceType code in cuVS with this PR. (Also the DistanceType code in RAFT has a Precomputed option that isn't available in cuvs - but is needed by cuml for dbscan.) This means that we have both the raft and cuvs DistanceType enum's in use with this change, with conversions between them.

This reverts commit cbb79ec.

dependencies.yaml

bdice

I took an early peek at the packaging side of this change, since this will probably be coming down close to freeze. Ping if you need help with any build issues / etc.

ci/release/update-version.sh

dependencies.yaml

We don't currently have cosine distance for ivf-pq (see rapidsai/cuvs#346) and we also don't have correlation distance support at all. re-add the metricprocessor code to handle this

cpp/cmake/thirdparty/get_cuvs.cmake

dantegd

Code changes and everything looks good to me, pending returning raft to branch-24.12

cjnolet

This is really great! I'm going to pre-approve just to get this in for the release, but don't forget to revert the explicit cmake pinning.

cjnolet · 2024-10-03T13:13:13Z

conda/environments/all_cuda-118_arch-x86_64.yaml

@@ -14,6 +14,7 @@ dependencies:
 - cudatoolkit
 - cudf==24.10.*,>=0.0.0a0
 - cupy>=12.0.0
+- cuvs==24.10.*,>=0.0.0a0


I don't think cuml needs a dependency on cuvs python today. We could keep this for the release, though, just to avoid additional changes.

I think you are correct here (we only need cuvs for the requirements/pyproject outputs for wheels). Let's file a follow-up PR once this one merges so we don't lose track. It can target 24.12.

Thankfully, cuvs is only ~350 kB: https://anaconda.org/rapidsai-nightly/cuvs/files

cpp/cmake/thirdparty/get_cuvs.cmake

cjnolet · 2024-10-03T13:14:22Z

cpp/include/cuml/cluster/kmeans.hpp

@@ -18,7 +18,7 @@

 #include <cuml/common/log_levels.hpp>

-#include <raft/cluster/kmeans_types.hpp>
+#include <cuvs/cluster/kmeans.hpp>


Looks great!

cjnolet · 2024-10-03T13:16:29Z

cpp/src/hdbscan/runner.h

@@ -24,8 +24,8 @@
 #include <cuml/cluster/hdbscan.hpp>
 #include <cuml/common/logger.hpp>

-#include <raft/cluster/detail/agglomerative.cuh>
-#include <raft/cluster/detail/mst.cuh>
+#include <raft/cluster/detail/agglomerative.cuh>  // build_dendrogram_host


Oh boy- detail APIs. Well need to make sure we add this to the tracker list. The single-linkage implementation from RAFT has been migrated to cuVS so we should be able to use that. No need to do that for 24.10, though.

cjnolet · 2024-10-03T13:17:28Z

cpp/src/kmeans/kmeans_mg_impl.cuh

@@ -1,818 +0,0 @@
-/*
- * Copyright (c) 2020-2024, NVIDIA CORPORATION.
- *


Beautiful to see this go. Eventually, we should start building distributed Python implementations right in cuVS for things like cuvs-dask (and cuvs-ray, etc...).

cjnolet · 2024-10-03T13:19:07Z

cpp/src/knn/knn.cu

+  // fully serial execution.
+  raft::resource::sync_stream_pool(handle);
+
+  if (input.size() > 1 || translations != nullptr) {


In the most ideal case, we should be able to formalize an "id_mapper" API at some point and have it used consistently for all index types in cuVS itself. That's a pretty trivial gather operation that could be accelerated.

cpp/CMakeLists.txt

cpp/cmake/thirdparty/get_cuvs.cmake

cpp/bench/CMakeLists.txt

cpp/test/CMakeLists.txt

readd the pin to check to see if changes in https://github.com/rapidsai/cuvs/compare/branch-24.10...benfred:static_lib2?expand=1 resolve wheel build issues.

bdice · 2024-10-03T17:36:02Z

conda/environments/all_cuda-118_arch-x86_64.yaml

@@ -14,6 +14,7 @@ dependencies:
 - cudatoolkit
 - cudf==24.10.*,>=0.0.0a0
 - cupy>=12.0.0
+- cuvs==24.10.*,>=0.0.0a0


I think you are correct here (we only need cuvs for the requirements/pyproject outputs for wheels). Let's file a follow-up PR once this one merges so we don't lose track. It can target 24.12.

Thankfully, cuvs is only ~350 kB: https://anaconda.org/rapidsai-nightly/cuvs/files

cpp/cmake/thirdparty/get_cuvs.cmake

bdice

I previously reviewed all the build/packaging changes, and am approving the new diff since my last look. I did not review the C++ code. Thanks for the hard work on this!

We need to do some planning on all the follow-up work that is needed to simplify cuVS and cuML. The build/packaging side of things is getting messy in both repositories with all the short term fixes.

Code freeze.

cjnolet · 2024-10-04T03:03:27Z

/merge

benfred and others added 10 commits July 16, 2024 14:59

Migrate from raft to cuvs for pairwise_distance and bfknn

2f2c55c

Merge branch 'branch-24.08' into cuvs

9116ae8

.

84bc77a

use cuvs::distance::DistanceType where possible

cbb79ec

Revert "use cuvs::distance::DistanceType where possible"

f653059

This reverts commit cbb79ec.

use stats from raft

a6f2c2a

Merge remote-tracking branch 'origin/branch-24.10' into cuvs

c2c9c04

use ivf-* from cuvs

95b9c14

Merge remote-tracking branch 'origin/branch-24.10' into cuvs

1934d40

Merge branch 'rapidsai:branch-24.10' into cuvs

0601412

github-actions bot added Cython / Python Cython or Python issue CMake CUDA/C++ labels Sep 26, 2024

benfred added 2 commits September 26, 2024 11:58

add libcuvs to dependencies.yaml

f30f933

Merge branch 'cuvs' of https://github.com/benfred/cuml into cuvs

69d2398

github-actions bot added conda conda issue ci labels Sep 26, 2024

dantegd reviewed Sep 26, 2024

View reviewed changes

dependencies.yaml Show resolved Hide resolved

benfred added 2 commits September 26, 2024 13:52

attempt to fix build error in CI

6c46624

Merge branch 'branch-24.10' into cuvs

3fb5479

cjnolet added the non-breaking Non-breaking change label Sep 27, 2024

benfred added 3 commits September 26, 2024 19:38

fix tsne

5380c17

Merge branch 'cuvs' of https://github.com/benfred/cuml into cuvs

dc1f7a6

fix ivf-fla

fdd18c5

bdice reviewed Sep 27, 2024

View reviewed changes

ci/release/update-version.sh Outdated Show resolved Hide resolved

dependencies.yaml Show resolved Hide resolved

benfred added 5 commits September 27, 2024 15:42

fix test_nearest_neighbors_rbc test for haversine distance

590865d

re-add MetricProcessor code

c7d1b0e

We don't currently have cosine distance for ivf-pq (see rapidsai/cuvs#346) and we also don't have correlation distance support at all. re-add the metricprocessor code to handle this

suggestions from code review

e8c1b18

fix dask pytests

bd58347

attempt to fix python build errors in CI

adb450a

Set rpath for cuvs

c12f1d5

cjnolet assigned benfred Oct 2, 2024

KyleFromNVIDIA and others added 7 commits October 2, 2024 11:12

Don't link Python modules against cuvs directly

f3edb8f

Remove superfluous cuvs::cuvs references

f906e79

Add cuvs rpath

585acd4

remove cuvs pin

1964a94

Merge branch 'cuvs' of https://github.com/benfred/cuml into cuvs

f1db388

updates to handle bfknn api changes

259256a

link cuvs statically in python wheels

4efd97a

dantegd requested changes Oct 3, 2024

View reviewed changes

cpp/cmake/thirdparty/get_cuvs.cmake Outdated Show resolved Hide resolved

benfred added 3 commits October 2, 2024 21:11

empty commit for ci

41daf01

empty commit for ci

31331d7

empty commit for ci

6bef472

dantegd reviewed Oct 3, 2024

View reviewed changes

cjnolet approved these changes Oct 3, 2024

View reviewed changes

benfred and others added 2 commits October 3, 2024 07:20

remove pin

3e88b8e

Merge branch 'branch-24.10' into cuvs

064cced

KyleFromNVIDIA reviewed Oct 3, 2024

View reviewed changes

cpp/CMakeLists.txt Show resolved Hide resolved

KyleFromNVIDIA previously requested changes Oct 3, 2024

View reviewed changes

cpp/cmake/thirdparty/get_cuvs.cmake Outdated Show resolved Hide resolved

cpp/bench/CMakeLists.txt Outdated Show resolved Hide resolved

cpp/test/CMakeLists.txt Outdated Show resolved Hide resolved

benfred added 3 commits October 3, 2024 10:21

re-add pin + suggestions from code review

3959c58

readd the pin to check to see if changes in https://github.com/rapidsai/cuvs/compare/branch-24.10...benfred:static_lib2?expand=1 resolve wheel build issues.

Merge branch 'cuvs' of https://github.com/benfred/cuml into cuvs

dc6de84

fix

461a271

bdice reviewed Oct 3, 2024

View reviewed changes

remove pin

f50b9d2

dantegd approved these changes Oct 4, 2024

View reviewed changes

bdice approved these changes Oct 4, 2024

View reviewed changes

rapids-bot bot merged commit 70fe526 into rapidsai:branch-24.10 Oct 4, 2024
62 of 63 checks passed

benfred deleted the cuvs branch October 4, 2024 03:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate to use cuVS for vector search #6085

Migrate to use cuVS for vector search #6085

benfred commented Sep 26, 2024 •

edited

Loading

bdice left a comment •

edited

Loading

dantegd left a comment

cjnolet left a comment

cjnolet Oct 3, 2024

bdice Oct 3, 2024

cjnolet Oct 3, 2024

cjnolet Oct 3, 2024

cjnolet Oct 3, 2024

cjnolet Oct 3, 2024

bdice Oct 3, 2024

bdice left a comment •

edited

Loading

cjnolet commented Oct 4, 2024

Migrate to use cuVS for vector search #6085

Migrate to use cuVS for vector search #6085

Conversation

benfred commented Sep 26, 2024 • edited Loading

bdice left a comment • edited Loading

Choose a reason for hiding this comment

dantegd left a comment

Choose a reason for hiding this comment

cjnolet left a comment

Choose a reason for hiding this comment

cjnolet Oct 3, 2024

Choose a reason for hiding this comment

bdice Oct 3, 2024

Choose a reason for hiding this comment

cjnolet Oct 3, 2024

Choose a reason for hiding this comment

cjnolet Oct 3, 2024

Choose a reason for hiding this comment

cjnolet Oct 3, 2024

Choose a reason for hiding this comment

cjnolet Oct 3, 2024

Choose a reason for hiding this comment

bdice Oct 3, 2024

Choose a reason for hiding this comment

bdice left a comment • edited Loading

Choose a reason for hiding this comment

cjnolet commented Oct 4, 2024

benfred commented Sep 26, 2024 •

edited

Loading

bdice left a comment •

edited

Loading

bdice left a comment •

edited

Loading