AZP: UCXX integration - tests + builds by Alexey-Rivkin · Pull Request #11473 · openucx/ucx

Alexey-Rivkin · 2026-05-20T11:42:51Z

What?

Add UCXX_build + UCXX_tests stages to the UCX PR pipeline. Each UCX PR builds UCXX conda packages, libucxx/ucxx wheels, and docs from rapidsai/ucxx, and runs the C++ (CPU+GPU), Python (GPU), and wheel tests.

Why?

Move UCXX CI from RAPIDS GitHub Actions onto UCX's Azure pipeline (mirrors upstream pr.yaml).

How?

Two runner scripts in buildlib/tools/ (build_ucxx.sh, test_ucxx.sh) + container images wrapping rapidsai/ci-conda and rapidsai/ci-wheel. Matrix: CUDA 12 + 13 × x86_64 + aarch64.

pentschev · 2026-06-08T07:13:59Z

+# Azure wrapper around rapidsai/ci-conda: chmod /opt/conda so the non-root UID Azure runs
+# steps as can use conda/python (rapidsai owns it as root); + adds gdb for stack capture.
+
+ARG BASE_IMAGE=rapidsai/ci-conda:26.06-latest


Since you've started, RAPIDS 26.06 was released and all ToT development is now happening for 26.08, including the images we depend on. I suggest targeting 26.08 throughout this PR as well.

c668a97
Thanks!

pentschev · 2026-06-08T07:24:57Z

+# Upstream ucxx header uses usleep() but omits <unistd.h>; undeclared on
+# newer gcc. Affects all C++ phases.


The fix rapidsai/ucxx#674 has been merged on main. If you switch to building main this should not be necessary anymore.

Will there be a new tag soon?

Not too soon, this is why I'd prefer to target main, but Yossi has a preference for stability at this time (understandable) so we may have to wait. The next tag should occur around July 16. Maybe instead of relying on specific tags we can test and target specific commits instead, such that we can do controlled upgrades? I fear keeping an older tag may diverge from RAPIDS CI updates, as you have seen there are several aspects that need to work in tandem (CI images, CI scripts in the project such as ucxx/ci, etc.).

Thanks for the tip! I went ahead and pinned it to the latest main SHA for now. That lets me drop both patches. Going forward, we can update the SHA or switch to a tag in a controlled manner as RAPIDS advances.
ac9f7b1

pentschev · 2026-06-08T07:25:07Z

+    # Upstream ucxx examples header uses usleep() but omits <unistd.h>;
+    # undeclared on newer gcc. Same patch as build_ucxx.sh.


pentschev · 2026-06-24T14:06:19Z

+      type: github
+      name: rapidsai/ucxx
+      endpoint: Mellanox-lab
+      ref: 33deb0b581b78027730e8ef86ed32efbb22d0dd8


This seems to pin to a specific commit, is this intended?

pentschev · 2026-06-24T14:10:09Z

+        - { name: x86_64_cuda12_py311, container: ucxx_rapidsai_ci_wheel_cuda12_gpu, libucxx_slice: x86_64_cuda12_py311, ucxx_slice: x86_64_cuda12_py311,
+            demands: ucx_gpu, rapids_cuda_version: '12.9.1', rapids_py_version: '3.11' }
+        - { name: x86_64_cuda13_py311, container: ucxx_rapidsai_ci_wheel_cuda13_gpu, libucxx_slice: x86_64_cuda13_py311, ucxx_slice: x86_64_cuda13_py311,
+            demands: ucx_gpu, rapids_cuda_version: '13.2.0', rapids_py_version: '3.11' }


RAPIDS has since bumped to CUDA 13.3 rapidsai/ucxx#679. I'm not sure what's the better way to keep this in sync. I suppose we'll need to have our ops team track UCXX CI when make such changes, at least while everything moves to the ucx org (i.e., when the release process is fully managed here).

@gforsyth thoughts?

yeah, we'll have to keep this updated during the transtion. We can track this in rapids build-engineering

Bumped the UCXX ref to current main and build images to CUDA 13.3

gleon99 · 2026-07-21T16:13:07Z

+            - checkout: ucxx
+              path: ucxx
+              retryCountOnTaskFailure: 5
+            - bash: bash $(UCX_DIR)/buildlib/tools/test_ucxx.sh build


[P1] Tests do not consume the UCX PR

This invokes UCXX's standard build scripts, but neither this job nor test_ucxx.sh builds or exports the checked-out UCX tree. The pinned UCXX recipes instead install UCX 1.18/1.19 packages, while this PR targets UCX 1.22. As a result, these jobs can pass without exercising any code from the UCX PR.

Crucial point!
Restructured the UCXX CI to build UCX from the PR checkout once per arch/CUDA pair in a dedicated stage. The UCXX build and test stages then reuse the generated artifacts. Verified that logs now show the correct UCX ver at every stage.

gleon99 · 2026-07-21T16:13:32Z

+      UCXX_DIR: $(Agent.BuildDirectory)/ucxx
+    jobs:
+      - ${{ each slice in parameters.slices }}:
+        - job: ucxx_tests_${{ slice.name }}


[P1] UCXX failures are not advisory

These are ordinary jobs with continueOnError defaulting to false, so a UCXX failure contributes a failed result to the main UCX PR pipeline. This contradicts the agreed requirement that UCXX failures never block UCX PRs.

Now all UCXX jobs run with continueOnError: true, so failures surface as warnings and never block the PR.

gleon99 · 2026-07-21T16:14:31Z

+# Guard catches upstream rewording (else the skip silently disappears).
+sed -i "s#--runslow#--runslow -k 'not test_client_shutdown'#" ci/run_python.sh
+grep -q "not test_client_shutdown" ci/run_python.sh \
+  || { echo "ERROR: test_client_shutdown skip did not apply to ci/run_python.sh" >&2; exit 1; }


[P2] Shutdown coverage is silently disabled

This rewrites the pinned UCXX test runner to exclude every test_client_shutdown variant from both conda and wheel testing. That can hide regressions this integration is intended to detect.

Those tests were very unstable, so I preffered not to introduce noise.
Now that all UCXX jobs run with continueOnError, we can safely run any tests - failures will be reported as warnings without blocking the PR.

gleon99 · 2026-07-21T16:15:30Z

+  printf '#!/bin/bash\necho "%s"\n' "$RAPIDS_BLD_OUTPUT_DIR" > "$HOME/.local/bin/$tool"
+  chmod +x "$HOME/.local/bin/$tool"
+done
+# Docs phase: override shims to point at the staged conda channels.


[P2] C++ docs channel shim is unused

The pinned UCXX build_docs.sh calls rapids-download-from-github for both its C++ and Python artifacts, so this rapids-download-conda-from-github shim is never used and both lookups resolve to PYTHON_CHANNEL_DIR.

Good catch, fixed.
The shim now overrides rapids-download-from-github and picks the channel by artifact name.
Also dropped the rapids-download-conda-from-github and rapids-download-wheels-from-github shims as unused.

gleon99 · 2026-07-23T08:46:41Z

@Alexey-Rivkin ready for re-review?

Alexey-Rivkin · 2026-07-23T08:50:54Z

@Alexey-Rivkin ready for re-review?

Not yet, sorry. We're currently hitting an infra limitation: several nodes are running out of disk space due to the RAPIDS images. I need to sort that out first.

The upstream rapidsai images are not writable by the non-root UID Azure runs steps as, and carry no gdb for crash stacks.

Run rapidsai/ucxx CI against the PR's UCX rather than released ucx packages, so the suites exercise the submitted UCP/UCS code. UCXX jobs are advisory and never block the UCX PR.

Every UCXX job rebuilt the same UCX, costing ~4 minutes each. Build it once per arch/CUDA pair and share it as an artifact.

The wheel test env downloads ~2 GB of cuda/rapids wheels per run.

UCXX needs nothing from Static_check, so its ~26 min just idled the UCXX jobs.

The pipeline runs these as `bash <path>`, which ignores the shebang, so the fail-loud guards were silently discarded.

The env is solved fresh each run; conda-forge moved to gcc 15/16, which rejects ucx_perftest's OpenMP 'master' under -Werror.

Alexey-Rivkin changed the title ~~AZP: add UCXX_tests stage to PR pipeline (Phase 1 plumbing)~~ AZP: add UCXX_tests stage to PR pipeline (Phase #1) May 20, 2026

Alexey-Rivkin force-pushed the ucxx-azure-tests branch 17 times, most recently from 05531f8 to 1def0bd Compare May 24, 2026 15:24

Alexey-Rivkin mentioned this pull request May 24, 2026

AZP: Build UCXX conda + wheel packages (Phase #2) #11484

Closed

Alexey-Rivkin force-pushed the ucxx-azure-tests branch 3 times, most recently from 1a79d8a to dec2673 Compare May 24, 2026 19:25

Alexey-Rivkin changed the title ~~AZP: add UCXX_tests stage to PR pipeline (Phase #1)~~ AZP: UCXX integration - tests + builds May 24, 2026

Alexey-Rivkin force-pushed the ucxx-azure-tests branch 7 times, most recently from 3b30b46 to e78b89e Compare May 25, 2026 03:24

Alexey-Rivkin marked this pull request as ready for review June 2, 2026 16:30

Alexey-Rivkin force-pushed the ucxx-azure-tests branch 4 times, most recently from 0a45412 to 24623e9 Compare June 2, 2026 21:00

pentschev reviewed Jun 8, 2026

View reviewed changes

Alexey-Rivkin force-pushed the ucxx-azure-tests branch from ac9f7b1 to 3cb3bed Compare June 9, 2026 08:26

Alexey-Rivkin requested a review from dpressle June 9, 2026 08:30

pentschev reviewed Jun 24, 2026

View reviewed changes

Alexey-Rivkin force-pushed the ucxx-azure-tests branch from 6e77ce4 to c8138b2 Compare June 25, 2026 20:24

pentschev mentioned this pull request Jul 2, 2026

Add UCX-Py testing #7412

Closed

dpressle previously approved these changes Jul 5, 2026

View reviewed changes

Alexey-Rivkin mentioned this pull request Jul 8, 2026

BUILD: add RAPIDS wrapper-image build tooling #11633

Draft

gleon99 reviewed Jul 21, 2026

View reviewed changes

Alexey-Rivkin dismissed dpressle’s stale review via f304e5c July 22, 2026 13:16

Alexey-Rivkin force-pushed the ucxx-azure-tests branch 2 times, most recently from 55b0c99 to edc2ad3 Compare July 22, 2026 17:38

Alexey-Rivkin marked this pull request as draft July 23, 2026 08:50

Alexey-Rivkin force-pushed the ucxx-azure-tests branch 3 times, most recently from f226d1d to a714b3b Compare July 23, 2026 16:01

Alexey-Rivkin added 7 commits July 27, 2026 08:57

BUILD: add rapidsai CI wrapper images for UCXX

0b12d01

The upstream rapidsai images are not writable by the non-root UID Azure runs steps as, and carry no gdb for crash stacks.

AZP: add UCXX build + test stages to the PR pipeline

5d126ee

Run rapidsai/ucxx CI against the PR's UCX rather than released ucx packages, so the suites exercise the submitted UCP/UCS code. UCXX jobs are advisory and never block the UCX PR.

AZP: build the PR's UCX once per pipeline run

ad107af

Every UCXX job rebuilt the same UCX, costing ~4 minutes each. Build it once per arch/CUDA pair and share it as an artifact.

AZP: persistent pip cache for the UCXX jobs

0e3fe54

The wheel test env downloads ~2 GB of cuda/rapids wheels per run.

AZP: start the UCXX stages after Basic_compile, not Static_check

3745c8e

UCXX needs nothing from Static_check, so its ~26 min just idled the UCXX jobs.

AZP: make the UCXX driver scripts fail on error

4ea0368

The pipeline runs these as `bash <path>`, which ignores the shebang, so the fail-loud guards were silently discarded.

AZP: pin the UCX-PR toolchain compiler to gcc 14

3f39051

The env is solved fresh each run; conda-forge moved to gcc 15/16, which rejects ucx_perftest's OpenMP 'master' under -Werror.

		# Upstream ucxx header uses usleep() but omits <unistd.h>; undeclared on
		# newer gcc. Affects all C++ phases.

		# Upstream ucxx examples header uses usleep() but omits <unistd.h>;
		# undeclared on newer gcc. Same patch as build_ucxx.sh.

Uh oh!

Conversation

Alexey-Rivkin commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

Why?

How?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gleon99 commented Jul 23, 2026

Uh oh!

Alexey-Rivkin commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Alexey-Rivkin commented May 20, 2026 •

edited

Loading