ci: Add nixl ep to new dl ci pipeline by lishapira · Pull Request #1802 · ai-dynamo/nixl

lishapira · 2026-06-22T12:02:35Z

Add nixl-ci-dl-gpu-ep job for nixl_ep elastic tests on dlcluster.
Integrates nixl_ep elastic tests into CI by adding elastic coverage to a
new nixl-ci-dl-gpu-ep Jenkins pipeline.

Changes:

CI: Create new test-dl-ep-matrix.yaml.
Build: Enable nixl_ep builds from CI and use an EP-local buildtype=release override to avoid CUDA -G/nvlink register-count failures.
test_ep.sh: EP CI script that runs elastic.py on 4 GPUs with no_expansion.json
and expansion_fault_contraction.json to exercise nixl_ep elastic
scale-up/scale-down. Test nvlink and rdma flows. The rdma flow only runs
when all four CX-7 NICs (mlx5_0..mlx5_3) report PORT_ACTIVE.
elastic.py: Add phase-local failure validation behind --validate-phase-failures
and change failure log wording.
Declare nixl_ep.topk_idx_t in the meta-dispatcher TYPE_CHECKING block so mypy
can see the runtime-exported attribute.

Summary by CodeRabbit

New Features
- Added a new DLCluster GPU EP test job and end-to-end elastic testing workflow.
- Added --validate-phase-failures for strict per-phase rank-failure validation.
- Enabled an optional EP build mode for GPU test builds.
Bug Fixes
- Improved elastic test failure reporting by clearly separating expected vs unexpected rank failures.
Chores
- Refreshed CI image tags across GPU and non-GPU pipelines and updated CI tag validation/reporting to cover the new EP matrix configuration.

copy-pr-bot · 2026-06-22T12:02:39Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-06-22T12:02:47Z

👋 Hi lishapira! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

…cluster Integrates nixl_ep elastic tests into CI by adding elastic coverage to a new nixl-ci-dl-gpu-ep Jenkins pipeline. Changes: CI: Create new test-dl-ep-matrix.yaml. Build: Enable nixl_ep builds from CI, install/copy DOCA GPUNetIO headers, skip nixl_ep when UCX GPU Device API is unavailable, and use an EP-local buildtype=custom override to avoid CUDA -G/nvlink register-count failures. test_ep.sh: EP CI script that runs elastic.py on 4 GPUs with no_expansion.json and expansion_fault_contraction.json to exercise nixl_ep elastic scale-up/scale-down. Test nvlink and rdma flows. elastic.py: Add phase-local failure validation behind --validate-phase-failures and change failure log wording. Declare nixl_ep.topk_idx_t in the meta-dispatcher TYPE_CHECKING block so mypy can see the runtime-exported attribute.

Give the EP CI job its own dedicated base + build_helper images, separate from the nixl-ci-dl-gpu job. Renamed in test-dl-ep-matrix.yaml: - nixl-ci-dl-gpu-base-... -> nixl-ci-dl-gpu-ep-base-... - build_helper_dl -> build_helper_dl_ep This isolates EP-only base-image changes (e.g. the DOCA GPUNetIO header staging added in this commit) from the existing DL pipeline; the two artifactory images are byte-identical today but can diverge later without affecting nixl-ci-dl-gpu. Remove EP-specific DOCA block in build.sh: 1. Delete duplicated DOCA install (exists in the base-image bootstrap). 2. Move DOCA header-copy into the base-image bootstrap, adjacent to the DOCA install, so PR builds inherit the staged headers for free. Bump CI_IMAGE_TAG to 20260622-2 in all five matrices so the base image rebuilds with the new bootstrap.

The new nixl-ci-dl-gpu-ep job runs test_ep.sh standalone, without the test_python.sh step that the old in-tree EP flow relied on. As a result `import nixl_ep` failed with ModuleNotFoundError because neither the dispatcher (nixl_ep/__init__.py) nor the compiled backend (nixl_ep_cu13) was reachable from the slurm container's Python: 1. The PR image's meson build installs nixl_ep_cu13 under ${INSTALL_DIR}/lib/python3/dist-packages, which is not on Python's default sys.path. Add that directory to PYTHONPATH so the dispatcher can importlib.import_module("nixl_ep_cu13") at runtime. 2. The PR image stages the nixl meta wheel in dist/ but never pip installs it, so the nixl_ep dispatcher package is not on sys.path. pip install dist/nixl-*none-any.whl mirrors test_python.sh and makes `import nixl_ep` resolve via the dispatcher. Other test_python.sh pre-test steps (tomlutil + pip install ., pytest / zmq deps, start_etcd_server) are intentionally skipped: the PR image already built nixl_ep_cu13, and elastic.py uses a local TCP rank_server plus torch TCPStore rather than pytest or etcd.

* test_ep.sh: change elastic test arguments; drop UCX_RNDV_THRESH=inf * examples/device/ep/meson.build: Replace ['buildtype=custom', 'optimization=3'] with ['buildtype=release'] so the debug override matches the release build flag bundle; Drop the ucx_gpu_device_api_available skip.

lishapira · 2026-06-24T10:30:47Z

/ok to test 92cccb7

lishapira · 2026-06-24T10:31:16Z

/build

NirWolfer · 2026-06-24T10:33:17Z

/ok to test 92cccb7

NirWolfer · 2026-06-24T12:47:36Z

/ok to test 92cccb7

NirWolfer · 2026-06-24T12:48:30Z

Looks good to me, i also see that the latest run of this pipeline passed: https://nbuprod.blsm.nvidia.com/nbu-swx-nixl-main/job/nixl-ci-dl-gpu-ep/3/

I will update the dispatcher pipeline once this is merged

…t-debug * test_ep.sh: drop the stale `kill -9 $ETCD_PID`; start_etcd_server is intentionally not invoked here. * build.sh: replace silent `|| true` on the DOCA GPUNetIO cp with an if/else that prints a failure message to stderr, and drop `2>/dev/null` so any cp/xargs error message reaches the log. * examples/device/ep/meson.build: gate `enable_fast_debug` on the EP-effective buildtype. CI (--buildtype=debug forces EP to release) now drops ENABLE_FAST_DEBUG, using NUM_CPU_TIMEOUT_SECS=100. * .ci/jenkins/lib/*-matrix.yaml: bump CI_IMAGE_TAG to 20260624-1.

lishapira · 2026-06-24T13:16:42Z

/build

NirWolfer · 2026-06-24T13:45:31Z

/ok to test d8f44f3

NirWolfer · 2026-06-24T16:11:56Z

All looks good, note that under the partition so far, we encountered an error on gb-nvl-118-compute04 and gb-nvl-118-compute05 specifically due to no IB found.

Currently there are 2 more attempts running to see if there are any other issues on that partition

… fast-debug override * test_ep.sh: only run the --disable-ll-nvlink (RDMA) elastic tests when all four CX-7 NICs (mlx5_0..mlx5_3) report PORT_ACTIVE. * test_ep.sh: use --timeout-ms 10000 for fault tolerance tests to make detection time shorter. * examples/device/ep/meson.build: revert the ep_effective_buildtype derivation introduced in the previous commit and restore the original enable_fast_debug = get_option('buildtype') != 'release'.

lishapira · 2026-06-25T08:32:34Z

/ok to test ba54abf

lishapira · 2026-06-25T08:32:47Z

/build

The nixl-ci-build-container nightly (nixl target, --build-type debug) fails at the EP device link: under -rdc=true, nixlPut compiles to ~215 registers and exceeds the EP kernels' __launch_bounds__ budget, so nvlink errors. ai-dynamo#1802 tried to dodge this by overriding the EP target to buildtype=release in a global debug build, but a per-target override of the composite `buildtype` is a silent no-op in meson - the target kept the global debug settings (optimization 0, -G), which disables --register-usage-level and inflates nixlPut's register count. Override the primitive options (optimization=3, debug=false) instead, which meson does apply per-target. This drops -G and optimizes the EP code so it links, while keeping -g for host-side debug symbols - the release-equivalent build ai-dynamo#1802 intended. Also pass --torch-versions in the EP wheel step of contrib/Dockerfile. ai-dynamo#1775 made --torch-versions mandatory with --build-nixl-ep and updated Dockerfile.manylinux, but left contrib/Dockerfile calling --build-nixl-ep alone - so every EP container build has errored out at the wheel step since. Mirror the manylinux default (WHL_TORCH_VERSIONS=2.11,2.12,2.13) so the build can complete.

The nixl-ci-build-container nightly (nixl target, --build-type debug) fails at the EP device link: under -rdc=true, nixlPut compiles to ~215 registers and exceeds the EP kernels' __launch_bounds__ budget, so nvlink errors. ai-dynamo#1802 tried to dodge this by overriding the EP target to buildtype=release in a global debug build, but a per-target override of the composite `buildtype` is a silent no-op in meson - the target kept the global debug settings (optimization 0, -G), which disables --register-usage-level and inflates nixlPut's register count. Override the primitive options (optimization=3, debug=false) instead, which meson does apply per-target. This drops -G and optimizes the EP code so it links, while keeping -g for host-side debug symbols - the release-equivalent build ai-dynamo#1802 intended.

…and set explicit EP meson overrides) (#1866) Fixes for nixl-ci-build-container flow ## What? 1. contrib/Dockerfile: add a new WHL_TORCH_VERSIONS build arg (default 2.11,2.12,2.13, mirroring Dockerfile.manylinux) and forward it to contrib/build-wheel.sh via --torch-versions when BUILD_NIXL_EP=true. 2. examples/device/ep/meson.build: extend the EP target's override_options from ['buildtype=release'] to ['buildtype=release', 'optimization=3', 'debug=false']. ## Why? After #1802 + #1775 landed, nixl-ci-build-container (debug, BUILD_NIXL_EP=true) has been failing in two distinct places: 1. Wheel step failure: #1775 added --torch-versions support to contrib/build-wheel.sh and updated Dockerfile.manylinux, but contrib/Dockerfile was still calling --build-nixl-ep alone. Every EP container build errored at the wheel step with --build-nixl-ep requires --torch-versions. 2. EP nvcc compile failure: #1802 sets nixl_ep target with override_options = ['buildtype=release'] so a global --buildtype=debug doesn't drop -G onto the EP nvcc line. In practice --device-debug still reached the EP .cu compiles on nixl-ci-build-containe compilation path.  ## Summary by CodeRabbit * **New Features** * Added support for configuring which Torch versions are included when building the wheel, with a sensible default set. * **Bug Fixes** * Improved EP builds in debug environments by forcing release-style compilation settings, helping avoid debug flags affecting CUDA compilation.

…and set explicit EP meson overrides) (#1866) (#1872) Cherry pick of #1866 Fixes for nixl-ci-build-container flow 1. contrib/Dockerfile: add a new WHL_TORCH_VERSIONS build arg (default 2.11,2.12,2.13, mirroring Dockerfile.manylinux) and forward it to contrib/build-wheel.sh via --torch-versions when BUILD_NIXL_EP=true. 2. examples/device/ep/meson.build: extend the EP target's override_options from ['buildtype=release'] to ['buildtype=release', 'optimization=3', 'debug=false']. After #1802 + #1775 landed, nixl-ci-build-container (debug, BUILD_NIXL_EP=true) has been failing in two distinct places: 1. Wheel step failure: #1775 added --torch-versions support to contrib/build-wheel.sh and updated Dockerfile.manylinux, but contrib/Dockerfile was still calling --build-nixl-ep alone. Every EP container build errored at the wheel step with --build-nixl-ep requires --torch-versions. 2. EP nvcc compile failure: #1802 sets nixl_ep target with override_options = ['buildtype=release'] so a global --buildtype=debug doesn't drop -G onto the EP nvcc line. In practice --device-debug still reached the EP .cu compiles on nixl-ci-build-containe compilation path. Co-authored-by: lishapira <lishapira@nvidia.com>

pull-request-size Bot added the size/L label Jun 22, 2026

github-actions Bot added the external-contribution label Jun 22, 2026

lishapira added 2 commits June 22, 2026 06:40

lishapira force-pushed the add_nixl_ep_to_new_dl_ci_pipeline branch from e23cc3b to aa664a1 Compare June 22, 2026 13:50

lishapira added 2 commits June 23, 2026 04:34

copy-pr-bot Bot temporarily deployed to SWX_AWS June 24, 2026 10:33 Inactive

copy-pr-bot Bot had a problem deploying to GITLAB June 24, 2026 10:33 Failure

NirWolfer previously approved these changes Jun 24, 2026

View reviewed changes

lishapira dismissed NirWolfer’s stale review via a6b8819 June 24, 2026 13:11

lishapira force-pushed the add_nixl_ep_to_new_dl_ci_pipeline branch from a6b8819 to d8f44f3 Compare June 24, 2026 13:15

copy-pr-bot Bot temporarily deployed to SWX_AWS June 24, 2026 13:45 Inactive

NirWolfer previously approved these changes Jun 24, 2026

View reviewed changes

lishapira dismissed NirWolfer’s stale review via ba54abf June 25, 2026 08:29

lishapira marked this pull request as ready for review June 25, 2026 09:00

itayalroy changed the title ~~ci: Add nixl ep to dl ci pipeline~~ ci: Add nixl ep to new dl ci pipeline Jun 28, 2026

itayalroy enabled auto-merge (squash) June 28, 2026 10:15

dpressle approved these changes Jun 28, 2026

View reviewed changes

ofirfarjun7 reviewed Jun 28, 2026

View reviewed changes

Comment thread .gitlab/test_ep.sh

Comment thread .ci/jenkins/lib/test-dl-ep-matrix.yaml

ofirfarjun7 approved these changes Jun 28, 2026

View reviewed changes

roiedanino approved these changes Jun 29, 2026

View reviewed changes

itayalroy merged commit 55bcf4d into ai-dynamo:main Jun 29, 2026
17 checks passed

coderabbitai Bot mentioned this pull request Jun 30, 2026

ci: fix DL EP test setup #1847

Merged

Alexey-Rivkin mentioned this pull request Jun 30, 2026

Build: pass --torch-versions in the EP wheel step of contrib/Dockerfile #1860

Closed

lishapira mentioned this pull request Jul 1, 2026

build: fix build-container flow (pass --torch-versions in Dockerfile and set explicit EP meson overrides) #1866

Merged

ovidiusm mentioned this pull request Jul 1, 2026

build: fix build-container flow (pass --torch-versions in Dockerfile and set explicit EP meson overrides) (#1866) #1872

Merged

This was referenced Jul 9, 2026

telemetry: don't unlink the producer's file when attaching fails #1826

Open

libfabric: update code owner list #1935

Merged

svc-nixl mentioned this pull request Sep 2, 2026

device api: Layer UCX device API from GPU API #2147

Merged

Uh oh!

Conversation

lishapira commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

copy-pr-bot Bot commented Jun 22, 2026

Uh oh!

github-actions Bot commented Jun 22, 2026

Uh oh!

lishapira commented Jun 24, 2026

Uh oh!

lishapira commented Jun 24, 2026

Uh oh!

NirWolfer commented Jun 24, 2026

Uh oh!

NirWolfer commented Jun 24, 2026

Uh oh!

NirWolfer commented Jun 24, 2026

Uh oh!

lishapira commented Jun 24, 2026

Uh oh!

NirWolfer commented Jun 24, 2026

Uh oh!

NirWolfer commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lishapira commented Jun 25, 2026

Uh oh!

lishapira commented Jun 25, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

lishapira commented Jun 22, 2026 •

edited

Loading

NirWolfer commented Jun 24, 2026 •

edited

Loading