Allow rndv-based PUT/GET protocol without error handling

[tvegas1]

What?

Allow RNDV, even if it does not fully support error handling.

Why?

Needed to enable GET and PUT rendezvous-based protocols.

Summary by CodeRabbit

• Bug Fixes
  • Improved connection and transfer reliability when using newer UCX versions by enabling additional error-handling support during setup.

[github-actions]

👋 Hi tvegas1! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 133d6d2e-16b1-4863-88ec-ee6cc049b9f0

📥 Commits

Reviewing files that changed from the base of the PR and between 0670972 and f4e716b.

📒 Files selected for processing (1)

• src/plugins/ucx/ucx_utils.cpp

---

📝 Walkthrough

Walkthrough

In the nixlUcxContext constructor, a version guard is added so that when the UCX version is 1.22 or higher, the RNDV_PIPELINE_ERROR_HANDLING configuration parameter is set to "y".

Changes

UCX RNDV Pipeline Error Handling

Layer / File(s)	Summary
Version-gated RNDV config src/plugins/ucx/ucx_utils.cpp	Adds a ucpVersion_ >= UCP_VERSION(1, 22) check in the nixlUcxContext constructor to set RNDV_PIPELINE_ERROR_HANDLING to "y".

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Poem

A bunny hops through versions with care,
When 1.22 arrives, pipelines beware!
Error handling flips to "y" with glee,
A single guard, simple as can be.
🐇✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title matches the PR’s goal of enabling RNDV-based PUT/GET support, even though the implementation detail is more specific.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[brminich]

/build

[svc-nixl]

👀 Investigating nixl-ci-dl-gpu build 1268…

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu · commit 0eb2c35f

The memReg code itself is a simple wrapper and hasn't been modified in a way relevant to this failure. The error originates entirely in the UCX/driver layer:

• ibv_reg_dmabuf_mr(...) failed: Unknown error 524 — error 524 is the Linux kernel ENOTSUPP code, meaning the RDMA/mlx5 driver rejected the dmabuf memory-region registration.
• This happens at the very first VRAM registration in the "Intra-agent memory transfer test: P-Thr=OFF, VRAM" — before any transfer runs.

This is an environment/driver issue on the specific SLURM node gb200-nvl4-ts2-93 (the mlx5_16 device's dmabuf support), not a regression introduced by commit 0eb2c35. The second parallel branch failed independently for a completely unrelated reason: a pip network read timeout downloading nvidia_cublas (542MB) from files.pythonhosted.org — also infrastructure, not code.

Neither failure is attributable to the PR's code changes.

Summary: Two parallel CI branches failed — the aarch64/ucx-v1.21.x GPU test failed on VRAM dmabuf memory registration, and the aarch64/ucx-master build failed on a pip network timeout.

Root cause: The primary GPU-test failure is an environment/driver issue, not a code bug: ibv_reg_dmabuf_mr(address=0x632200000, length=67108864, access=0x10000f) failed: Unknown error 524. Error 524 is the Linux kernel ENOTSUPP — the mlx5_16 RDMA device on node gb200-nvl4-ts2-93 does not support (or is not configured for) dmabuf-based CUDA/VRAM memory registration. NIXL's memReg() simply propagates the failure from ucp_mem_map → the test aborts before any transfer. The second branch failure is unrelated infrastructure flakiness: pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out while downloading nvidia_cublas-13.1.0.3 (542 MB at ~225 kB/s).

Implicated commit: none — not attributable to 0eb2c35 / PR #1854. The implicated file (ucx_utils.cpp memReg) is unchanged wrapper code; the errors come from the UCX/RDMA driver stack and the package mirror.

File: src/plugins/ucx/ucx_utils.cpp:565 (the ucp_mem_map call that surfaces the driver error; not the root cause)

Suggested fix:

• For the dmabuf error: this is a node/driver problem. Verify dmabuf support on gb200-nvl4-ts2-93 (kernel PeerMemory/dmabuf config for the mlx5_16 NIC, MOFED/nv_peer_mem or dmabuf driver version). Consider retrying on a different node or setting UCX_IB_REG_METHODS/disabling dmabuf (e.g. UCX_MEM_REG_DMABUF=n / rely on peer-memory) if the fabric can't do dmabuf. This is not a code fix.
• For the pip timeout: increase pip's network resiliency in CI (pip install --timeout=120 --retries=5) and/or use a local/artifactory PyPI mirror to avoid the slow files.pythonhosted.org path. Re-run the job to confirm both are transient/environmental.

Related: none

[svc-nixl]

👀 Investigating nixl-ci-gpu build 2467…

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit 0eb2c35f

The logs clearly show this is an infrastructure failure, not a code/test failure. Let me analyze the timeline.

The key events:

• Build succeeded — the container image was built, nixlbench compiled and linked cleanly, and the image was pushed to the registry.
• Then in the "Allocate Environment" stage, both parallel branches (ucx-v1.21.x and ucx-master) tried to allocate a Slurm GPU node via scctl/salloc.

Looking at the timestamps around the allocation:

• 07:12:36 — salloc submitted for nixl-ci-master-2467, job 60138 "queued and waiting for resources"
• 08:12:22/08:12:45 — after ~1 hour of waiting, salloc: error: Unable to allocate resources: Connection timed out

The --immediate=3600 flag (3600 seconds = 1 hour) means Slurm was told to give up if it couldn't allocate within an hour. Both jobs sat in the mizu partition queue for the full hour and were rejected with "Connection timed out."

This is a cluster resource / connectivity problem, not a defect introduced by the PR. There is no test that ran and failed, no stack trace, no source-level error. The --gres=gpu:4, --mincpus=24, --mem=128G requests never got scheduled.

No source file or commit is implicated — the actual nixl build/test never executed. This is not a hang inside application code either; it's Slurm queue starvation / a control-plane connectivity timeout on the scctl head node.

Summary: GPU CI (build #2467) failed during the "Allocate Environment" stage — Slurm salloc could not obtain a GPU node and timed out after the 1-hour immediate window.

Root cause: Infrastructure/resource-availability failure, not a code defect. Both parallel branches (ucx-v1.21.x, ucx-master) submitted salloc to the mizu partition (--gres=gpu:4 --mincpus=24 --mem=128G --immediate=3600). Both jobs (60137, 60138) sat "queued and waiting for resources" for ~1 hour and were rejected with salloc: error: Unable to allocate resources: Connection timed out. The container image built and nixlbench compiled/linked/installed successfully beforehand — no test ever ran. This is Slurm queue starvation / a control-plane connectivity timeout on the scctl head node, unrelated to commit 0eb2c35 / PR #1854.

Implicated commit: unknown (not code-related — the failure is in the CI Slurm allocation step, which runs before any code from the PR is exercised)

File: unknown (no source file involved; failure is in the Jenkins slurm.allocation pipeline step against the mizu partition)

Suggested fix: This is a transient infra failure — retry the build. If it recurs: (1) check mizu partition health/availability and scctl head-node connectivity (the error is a connection timeout, suggesting the Slurm controller was unreachable or overloaded, not merely busy nodes); (2) consider adding a retry/backoff around the slurm.allocation step or raising immediateTimeout only if the cluster is genuinely just contended; (3) confirm GPU nodes with gpu:4 / 24 CPU / 128G capacity are online in mizu. Do not attribute this to PR #1854.

Related: none found (no issue/PR search performed — this is a self-evident infrastructure allocation timeout).

[svc-nixl]

👀 Investigating nixl-ci-dl-gpu build 1275…