Skip to content

CONTRIB: pin spcx-plugin to v0.2.x branch instead of master (v1.5.0 release) - #2014

Merged
roiedanino merged 1 commit into
ai-dynamo:mainfrom
roiedanino:use-spcx-plugin-v0.2.x-branch
Aug 3, 2026
Merged

roiedanino merged 1 commit into
ai-dynamo:mainfrom
roiedanino:use-spcx-plugin-v0.2.x-branch

Conversation

@roiedanino

@roiedanino roiedanino commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

What?

Change EXT plugin ref to v0.2.x release branch

Why?

Currently, it's pinned to master which aligned with UCX master while NIXL release uses v1.22.x,
v0.2.x branch of the plugin is aligned with UCX v1.22.x plugin API

Summary by CodeRabbit

  • Bug Fixes
    • Updated the default UCX SPCX plugin version used during container builds to the v0.2.x release line.
    • Build help and options now reflect the updated default when no version is specified.

…elease)

Signed-off-by: Roie Danino <rdanino@nvidia.com>
@roiedanino roiedanino self-assigned this Jul 30, 2026
@roiedanino
roiedanino requested a review from a team as a code owner July 30, 2026 13:55
@copy-pr-bot

copy-pr-bot Bot commented Jul 30, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown

👋 Hi roiedanino! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 539f9792-4bc4-4fbb-b488-b758995c5ed2

📥 Commits

Reviewing files that changed from the base of the PR and between 625f62a and c12fad7.

📒 Files selected for processing (1)
  • contrib/build-container.sh

📝 Walkthrough

Walkthrough

The build container script changes the default UCX SPCX plugin reference from main to v0.2.x.

Changes

UCX SPCX plugin reference

Layer / File(s) Summary
Update default plugin reference
contrib/build-container.sh
The default UCX_SPCX_PLUGIN_REF value is changed to v0.2.x, affecting plugin bundling and displayed option defaults.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Suggested reviewers: ofirfarjun7

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly states the main change: pinning the spcx-plugin ref from master to v0.2.x.
Description check ✅ Passed The description covers What and Why and matches the template; the optional How section is acceptable to omit.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@NirWolfer

Copy link
Copy Markdown
Contributor

/build

@roiedanino
roiedanino enabled auto-merge (squash) July 30, 2026 14:12
@roiedanino

Copy link
Copy Markdown
Contributor Author

/build

@roiedanino
roiedanino requested a review from gleon99 August 2, 2026 07:11
@svc-nixl

svc-nixl commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agentnixl-ci-gpu · commit 050b3f9c

TL;DR: The "Run Nixlbench tests" stage was killed (exit 143) after hanging ~24 minutes in UCCL backend teardown; the root cause is that PR #2000's skip of the flaky UCCL tests (issue #1999) was only applied to the ETCD-based block, leaving the ASIO-based UCCL block active — comment it out too.

Full analysis

Summary: Jenkins stage 299 ("Run Nixlbench tests", v1.22.x variant) was aborted with exit code 143 after the UCCL ASIO benchmark hung during engine shutdown.

Root cause: A hang, not a slow test. The last application output was the UCCL READ/VRAM→VRAM benchmark printing its result at 07:43:49.213, then "Destroying Engine..." → "Engine destroyed" and [WARN ... event_loop rdma/epoll_client.h:207] Error/HUP on connection: 1.1.101.2:37495 at 07:43:49.780. After that there is a ~24-minute silence (the largest gap by far, covering essentially the whole 1560 s stage duration) until Jenkins sent SIGTERM at 08:07:29.793. The nixlbench UCCL processes never exited, so parallel --halt now,fail=1 in run_nixlbench_two_workers_asio blocked forever. This is the known UCCL nixlbench shutdown hang tracked in issue #1999. The mitigation (PR #2000) disabled the UCCL tests — but only the ETCD UCCL loop (lines 124–132) was commented out; the ASIO UCCL loop (lines 95–103) was left enabled and is what hung here.

Implicated commit: 7a244b9 "CI: Skip UCCL nixlbench tests (#2000)" by ovidiusm (incomplete skip); the underlying hang is UCCL backend teardown per issue #1999.

File: .gitlab/test_nixlbench.sh:95-103 (the still-active --backend UCCL ASIO loop)

Suggested fix: Disable/comment out the ASIO UCCL block at lines 95–103 the same way the ETCD UCCL block (122–132) was, referencing issue #1999. As a defensive measure, also add a per-invocation timeout (e.g. wrap the parallel call in timeout 120) so a UCCL teardown hang fails fast instead of consuming the whole stage wall-clock and causing a SIGTERM abort. Do not simply raise the stage time limit — the process is hung, not slow.

Related: Issue #1999 (UCCL nixlbench test hangs); PR #2000 / commit 7a244b9 (partial skip).

@Alexey-Rivkin

Copy link
Copy Markdown
Contributor

/ok to test c12fad7

@roiedanino
roiedanino merged commit 1203802 into ai-dynamo:main Aug 3, 2026
20 checks passed
@roiedanino
roiedanino deleted the use-spcx-plugin-v0.2.x-branch branch August 3, 2026 11:57
NirWolfer added a commit to NirWolfer/nixl that referenced this pull request Aug 10, 2026
The spcx section said the flag requires --dockerfile
contrib/Dockerfile.manylinux, which this branch no longer enforces: the
default contrib/Dockerfile consumes it too, and only the wheel bundling
stays manylinux-only.

Also correct the plugin ref default, which has been v0.2.x since ai-dynamo#2014.

Signed-off-by: NirWolfer <nwolfer@nvidia.com>
NirWolfer added a commit to NirWolfer/nixl that referenced this pull request Aug 18, 2026
The spcx section said the flag requires --dockerfile
contrib/Dockerfile.manylinux, which this branch no longer enforces: the
default contrib/Dockerfile consumes it too, and only the wheel bundling
stays manylinux-only.

Also correct the plugin ref default, which has been v0.2.x since ai-dynamo#2014.

Signed-off-by: NirWolfer <nwolfer@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants