Cleanup pairwise distances

[jcrist]

• Removes usage of legacy input validation and CumlArray/SparseCumlArray. Part of Transition to new array input validation system #7428.
• Simplifies code flow, making the sparse_pairwise_distances duplicative and unneeded.
• Deprecates the sparse_pairwise_distances function. It's unclear why this was ever made public - pairwise_distances can do everything it can, and sklearn lacks a sparse_pairwise_distances function.
• Releases the GIL in libcuml calls.
• Fixes a bug in input reflection for nan_euclidean_distances. Previously this function failed to reflect at all, always returning cupy outputs.
• Adds a missing copy parameter to nan_euclidean_distances, improving sklearn compatibility.
• Greatly increases test coverage of nan_euclidean_distances.
• Standardizes error and warning messages to better match sklearn
• Increases test coverage for error handling and warnings.
• Deprecates the metric_arg to pairwise_distances, in favor of the existing kwds option. For minkowski, users should pass in p instead of metric_arg. Fixes [BUG] cuml pairwise_distances API differs from sklearn #4605.
• Adds error handling for unknown parameters passed to kwds. This caught a bug in the tests, where a non-existent output_type keyword argument was silently being ignored.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 46c673fd-2251-4a03-be17-1593933b2b0e

📥 Commits

Reviewing files that changed from the base of the PR and between ea934d9 and 4ea63d0.

📒 Files selected for processing (2)

• python/cuml/cuml/metrics/pairwise_distances.pyx
• python/cuml/tests/test_metrics.py

🚧 Files skipped from review as they are similar to previous changes (2)

• python/cuml/cuml/metrics/pairwise_distances.pyx
• python/cuml/tests/test_metrics.py

---

📝 Walkthrough

Summary by CodeRabbit

Release Notes

• New Features

  • Added a copy parameter to nan_euclidean_distances to control whether input data may be mutated.

• Deprecations

  • sparse_pairwise_distances is deprecated; use pairwise_distances for sparse inputs.
  • metric_arg usage is deprecated; pass metric parameters via keyword arguments.

• Bug Fixes

  • Improved distance computation consistency across dense and sparse inputs, including robust handling of missing values and squared/sqrt behavior.

• Tests

  • Expanded coverage for warnings, invalid inputs/metrics, deprecations, and sparse/dense output behavior.

Walkthrough

Refactors pairwise_distances.pyx to align with sklearn's **kwds-based API: deprecates the metric_arg parameter in favor of keyword arguments, unifies dense/sparse dispatch via _determine_metric, rewrites nan_euclidean_distances with a new copy parameter and native CuPy missing-value handling, reduces sparse_pairwise_distances to a FutureWarning deprecation shim, consolidates RussellRao boolean coercion via a shared _ensure_boolean helper used by KernelDensity, and expands test coverage across validation, output types, and sparse operations.

Changes

Pairwise distances API refactor: dense/sparse unification and nan_euclidean rewrite

Layer / File(s)	Summary
Imports and _determine_metric refactor python/cuml/cuml/metrics/pairwise_distances.pyx	Replaces scipy/numpy/pandas/cudf imports with cupyx.scipy.sparse and DataConversionWarning; rewrites _determine_metric to select dense vs sparse metric enum maps via an is_sparse flag with updated error messaging.
nan_euclidean_distances rewrite with copy parameter python/cuml/cuml/metrics/pairwise_distances.pyx	Adds @reflect and copy keyword argument; updates docstring; rewrites body to use check_array(ensure_all_finite="allow-nan"), CuPy missing-value masks via cp.isnan, delegate sqeuclidean call to pairwise_distances, then scale/clip/zero-diagonal/sqrt the result.
pairwise_distances kwds-based API and sparse_pairwise_distances deprecation python/cuml/cuml/metrics/pairwise_distances.pyx	Removes explicit metric_arg; handles it via **kwds with FutureWarning; validates unknown kwds; enforces sparse/dense mixing rules; checks feature dims; determines DistanceType; conditionally boolean-converts data; dispatches to dense or sparse C++ backend by dtype; replaces dedicated sparse implementation with deprecation shim that forwards to pairwise_distances.
KernelDensity RussellRao coercion consolidated python/cuml/cuml/neighbors/kernel_density.pyx	Removes _coerce_russellrao_binary helper and warnings import; imports _ensure_boolean from cuml.metrics.pairwise_distances; updates KernelDensity.fit and score_samples RussellRao paths to call _ensure_boolean.
Test imports and sparse deprecation coverage python/cuml/tests/test_metrics.py	Switches sparse generation to cupyx.scipy.sparse as cp_sp; adds DataConversionWarning import; adds test_sparse_pairwise_distances_deprecated asserting FutureWarning and validating results; removes now-unnecessary pytest.mark.filterwarnings for russellrao warnings.
pairwise_distances validation test suite python/cuml/tests/test_metrics.py	Introduces granular tests for unsupported metrics, dimension mismatches, sparse/dense mixing errors, boolean-conversion warnings, metric_arg deprecation, keyword-argument forwarding, and unknown parameters.
nan_euclidean_distances test expansion python/cuml/tests/test_metrics.py	Expands coverage to verify copy semantics and mutation behavior, missing_values handling, and squared option; compares against sklearn and exercises pairwise_distances with corresponding parameters.
Output-type test refactoring python/cuml/tests/test_metrics.py	Removes use_global parameterization; refactors sparse CSR generation to cp_sp.random(...).tocsr(); migrates sparse corner-case tests from sparse_pairwise_distances to pairwise_distances with ref_pairwise_distances_sparse comparisons; updates output-type assertions for cudf.DataFrame, numpy.ndarray, and cp.ndarray.
Sparse sklearn-comparison and output-type tests python/cuml/tests/test_metrics.py	Renames sparse sklearn-comparison test; swaps sparse_pairwise_distances calls to pairwise_distances with updated references; refactors sparse output-type testing to use cp_sp.random and cuml.using_output_type(output_type) context, with type assertions for returned outputs.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• rapidsai/cuml#8065: Substantially refactors the dense implementations of nan_euclidean_distances and pairwise_distances in the same pairwise_distances.pyx file, including new validation and input-normalization behavior that this PR builds upon.

Suggested labels

bug, non-breaking

Suggested reviewers

• betatim

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Title check	❓ Inconclusive	The title 'Cleanup pairwise distances' is vague and generic, using non-descriptive terms that don't convey the specific nature of the substantial refactoring and API changes made.	Use a more specific title such as 'Refactor pairwise distances to align with sklearn API, deprecate sparse_pairwise_distances and metric_arg' to better reflect the main objectives.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description check	✅ Passed	The pull request description is comprehensive and well-related to the changeset, covering legacy input validation removal, sparse function deprecation, GIL release, bug fixes, and new test coverage.
Linked Issues check	✅ Passed	The PR fully addresses issue #4605 by replacing metric_arg with kwds-based parameter passing, adding error handling for unknown parameters, and achieving sklearn API compatibility.
Out of Scope Changes check	✅ Passed	All changes are directly related to the stated objectives: removing CumlArray usage, deprecating sparse_pairwise_distances, fixing nan_euclidean_distances reflection, and standardizing the API to match sklearn.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@python/cuml/cuml/metrics/pairwise_distances.pyx`:
- Around line 329-338: The variable `p` declared as `cdef double p` is left
uninitialized and only conditionally assigned when `metric_arg` is in kwds or
when metric equals "minkowski". For all other metrics, the uninitialized
variable is passed to C++ functions (at lines 448, 466, 480, and 493), resulting
in undefined behavior. Initialize `p` with a default value (such as 2.0, the
standard Minkowski default) at the point of declaration to ensure it always has
a defined value regardless of which code path is taken.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ede423a8-c02f-4c52-ad21-815c17cebd60

📥 Commits

Reviewing files that changed from the base of the PR and between 98a26b7 and ea934d9.

📒 Files selected for processing (3)

• python/cuml/cuml/metrics/pairwise_distances.pyx
• python/cuml/cuml/neighbors/kernel_density.pyx
• python/cuml/tests/test_metrics.py

[csadorf]

Approved with one change recommendation.

[csadorf]

I think we should try to exhaustively document all additional metric-specific parameters as part of the metric doc-string or alternatively redirect to the right documentation. Otherwise it feels like we are just mentioning an example here, but are actually not fully documenting these additional parameters.

[jcrist]

sklearn doesn't document these fully either (you have to read the code to see what would actually work). My intent here was mainly to make our implementation compatible with sklearn, not to fully document all options. I'll open this as a followup issue if anyone wants to take it on, but want to get this in as is for now.

[csadorf]

Perfectly fine to handle this as a potential follow-up. I’m happy to follow sklearn’s lead on API behavior where it makes sense and avoid arbitrary deviations, but I do think it’s important to view cuML’s API documentation independently. We’re serving our users, so “sklearn does this too” shouldn’t by itself be the bar for whether the docs are clear enough.

[jcrist]

/merge