Support recent hdbscan

[jcrist]

This:

• Removes an upper bound from hdbscan in our tests. Previously we'd pinned to test against version 0.8.39, which means we'd missed several recent hdbscan releases. Now our "latest" deps tests will always use the latest release, ensuring we notice upstream breaking changes. We also test against the previous pinned version (in "oldest" test runs), as well as one intermediate version.
• Updates condensed_tree_ to work with recent hdbscan
• Updates our tests to run on valid combinations of hdbscan and sklearn
• Removes one newly failing test that I'd claim is bad (it previously didn't test anything on any run, now it sometimes tests things, but fails - deleting it for now made more sense than xfailing IMO, we have other existing tests that hit the same functionality).

Fixes #7955.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 89308b6e-416c-49f7-a315-258d9d5835a4

📥 Commits

Reviewing files that changed from the base of the PR and between 3fe69ea and dafcf4b.

📒 Files selected for processing (1)

• python/cuml/cuml/cluster/hdbscan/hdbscan.pyx

🚧 Files skipped from review as they are similar to previous changes (1)

• python/cuml/cuml/cluster/hdbscan/hdbscan.pyx

---

📝 Walkthrough

Summary by CodeRabbit

• Chores
  • Relaxed hdbscan dependency constraints across environments and packaging to allow newer versions (>=0.8.39) and moved specific pins into CI matrices.
• Tests
  • Conditionalized test execution based on installed hdbscan/sklearn versions and removed a redundant comparative test.
• Compatibility
  • Updated runtime behavior to support multiple hdbscan releases and handle newer hdbscan versions gracefully.

Walkthrough

Loosened hdbscan version pins across conda and project configs, moved hdbscan version control into CI matrices, added runtime hdbscan version detection in the Cython HDBSCAN wrapper to support pre-/post-0.8.41 CondensedTree signatures, and adjusted tests (guarded skip; removed one test).

Changes

Cohort / File(s)	Summary
Conda Environments conda/environments/all_cuda-129_arch-aarch64.yaml, conda/environments/all_cuda-129_arch-x86_64.yaml, conda/environments/all_cuda-131_arch-aarch64.yaml, conda/environments/all_cuda-131_arch-x86_64.yaml	Changed hdbscan>=0.8.39,<0.8.40 → hdbscan>=0.8.39.
Dependency Management dependencies.yaml	Removed hdbscan from test_python common list; added matrix-specific entries: hdbscan==0.8.39 (oldest), hdbscan==0.8.41 (intermediate), and hdbscan>=0.8.39 for the scikit_learn matrix.
Project Configuration python/cuml/pyproject.toml	Changed test optional dependency hdbscan>=0.8.39,<0.8.40 → hdbscan>=0.8.39.
HDBSCAN Wrapper (Cython) python/cuml/cuml/cluster/hdbscan/hdbscan.pyx	Added hdbscan_at_least(min_version) helper; branch condensed_tree_ behavior to call hdbscan.plots.CondensedTree with signatures conditional on installed hdbscan version (pre-/post-0.8.41).
Tests python/cuml/tests/test_hdbscan.py	Replaced unconditional module skip with a guarded import/version check skipping when hdbscan < 0.8.41 alongside sklearn >= 1.8.0; removed test_approximate_predict_circles.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• Update to 26.06 #7879 — Updates to hdbscan version constraints in conda envs and dependency files (touches same files/entries as this change).

Suggested reviewers

• gforsyth
• msarahan
• betatim

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'Support recent hdbscan' directly and clearly summarizes the main change: enabling compatibility with recent hdbscan releases by removing version constraints and updating implementation details.
Description check	✅ Passed	The description is directly related to the changeset, explaining the removal of upper bounds on hdbscan, updates to condensed_tree_ implementation, test adjustments, and removal of a failing test.
Linked Issues check	✅ Passed	The PR fully addresses issue #7955 by: (1) supporting the new CondensedTree constructor signature via version-aware branching in hdbscan.pyx, (2) maintaining backward compatibility with older versions, and (3) removing upper bounds to expose upstream breaking changes.
Out of Scope Changes check	✅ Passed	All changes are scoped to hdbscan compatibility: dependency version constraints are loosened/updated, the condensed_tree_ implementation is made version-aware, tests are updated for valid hdbscan/sklearn combinations, and one unreliable test is removed.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@python/cuml/cuml/cluster/hdbscan/hdbscan.pyx`:
- Around line 868-872: The code passes a CumlArray (self.labels_) directly into
hdbscan.plots.CondensedTree which expects a NumPy array; convert/normalize
labels_ to host NumPy before constructing the CondensedTree (use the existing
pattern via self._get_raw_data_cpu(self.labels_) or equivalent conversion) so
call hdbscan.plots.CondensedTree(self._condensed_tree,
self._get_raw_data_cpu(self.labels_)).

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 481ca140-678a-47c8-8c69-84e805343222

📥 Commits

Reviewing files that changed from the base of the PR and between b4eab6a and a7ff0f9.

📒 Files selected for processing (8)

• conda/environments/all_cuda-129_arch-aarch64.yaml
• conda/environments/all_cuda-129_arch-x86_64.yaml
• conda/environments/all_cuda-131_arch-aarch64.yaml
• conda/environments/all_cuda-131_arch-x86_64.yaml
• dependencies.yaml
• python/cuml/cuml/cluster/hdbscan/hdbscan.pyx
• python/cuml/pyproject.toml
• python/cuml/tests/test_hdbscan.py

[dantegd]

Change looks good to me

[jcrist]

/merge

[jameslamb]

great!