[Python] Reduce latency of NIXL import

[ovidiusm]

What?

Speedup NIXL wheel import time

How?

Slowest dependency import was profiled to be PyTorch.

Changes:

• Import torch only as needed. Not all projects using nixl use torch.
• Lazy import for torch tensor type.
• Rely on TYPE_CHECKING to import torch in IDEs and static analysis tools.
• Use a faster way to detect CUDA major version, relying on alternative CUDA wrappers (imported PyTorch, CUDA bindings, CuPy, non-imported PyTorch)
• Refactored duplicate code between nixl and nixl.ep modules to remove repetition

Summary by CodeRabbit

• New Features

  • Improved automatic selection of CUDA-enabled Python backends based on the detected CUDA version.

• Bug Fixes

  • Reduced startup issues by avoiding unnecessary GPU library imports during module loading.
  • Made backend selection more reliable when CUDA-related packages are already available or installed in different combinations.
  • Updated error messages to better reflect the detected CUDA version when a matching backend is missing.

[github-actions]

👋 Hi ovidiusm! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[ovidiusm]

/build

[svc-nixl]

👀 Investigating nixl-ci-dl-gpu build 1359…

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu · commit da303d04

Those sibling jobs don't exist as separate Jenkins jobs (the parallel branches run within the same nixl-ci-dl-gpu pipeline). The exit code 134 abort came from one of the parallel branches within build #1359, but the console log I retrieved is the DL-GPU/nixlbench branch, which passed entirely.

Let me reconsider the fetched log more carefully. The abort must have occurred in a parallel branch whose output was interleaved or in a branch (e.g., "v1.21.x" — note SLURM job 1483178 for v1.21.x and 1483185 for master were both stopped at the end). There were two parallel test branches: master and v1.21.x. The log I fetched is the master branch (container tag ...master:1359), which passed. The v1.21.x branch (UCX v1.21.x) is the other parallel task — and it's the one that likely aborted with 134, since PR #1831/#1874 note UCX/sanitizer instability. I don't have that branch's log.

Given my rules: I fetched a log successfully, but it shows a passing branch and does not contain the abort. I must not fabricate a root cause for the v1.21.x branch's SIGABRT without its log. I'll report what the evidence supports and flag the missing branch log.

Summary: Jenkins nixl-ci-dl-gpu #1359 failed with exit code 134 (SIGABRT) reported from one of the parallel test branches; the pipeline aborted.

Root cause: Undetermined from available evidence. The log I retrieved is the master/nixlbench parallel branch (container tag nixl-ci-dl-gpu-test-master:1359, SLURM job 1483185), and it ran to completion cleanly — every stage passed (C++ gtests 134/134, test_plugin "TEST PASSED", nixl_test "UCX Transfer Success", POSIX/GUSLI/Azure tests all OK, and all nixlbench READ/WRITE runs produced results). The fatal exit code 134 was raised at the pipeline aggregation level (parallel task failed with msg: ... script returned exit code 134), meaning the abort occurred in the other parallel branch (the UCX v1.21.x branch, SLURM job 1483178), whose console output is not present in the log I fetched. Exit 134 = SIGABRT (128+6): a process hit abort() — an uncaught C++ exception, failed assertion, or a sanitizer/UCX teardown abort — not a wall-clock timeout (there are no timing gaps; the branch I have was continuously active).

Implicated commit: unknown (the aborting branch's log was not available; cannot attribute).

File: unknown — the SIGABRT did not originate in the fetched branch.

Suggested fix: Retrieve the console log for the failing parallel branch — the UCX v1.21.x branch (SLURM job 1483178), not the master branch (1483185) that I inspected. That branch's log will contain the abort/assertion/backtrace or the specific test binary that returned 134. Note commit e286b8f7 ("ci: export container on test failure for crash debugging") should have exported a core/container on this crash — pull that artifact and run gdb/coredumpctl on it. Given PR #1895 only changes Python import latency, first confirm whether the abort reproduces on main (possible flaky UCX teardown abort, cf. #1874 skipping ucx_backend_multi under sanitizers and #1831) before attributing it to this PR.

Related: #1895 (the PR under test), #1874 (ci: skip ucx_backend_multi under TSan; lower sanitizer timeout), #1831 (ci: stop non-etcd sanitizer stages spawning etcd/gRPC thread), #1800 (ci: export container on test failure for crash debugging).

If you can point me to the v1.21.x branch console log (or the exported crash container/core dump) for build #1359, I can pinpoint the aborting test and the exact file:line.

[svc-nixl]

👀 Investigating nixl-ci-gpu build 2586…

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit da303d04

I now have a complete picture. Let me do the final timing math to distinguish "hang" from "legitimately slow."

The pytest-timeout is 5.0 seconds per test. Looking at the log, each single nixl_agent(...) construction with a UCX backend consistently emits Backend UCX was instantiated and takes ~11 seconds each — this pattern repeats uniformly 18 times across the whole run (13:00:42 → 13:04:09, one every ~11s). There is no large silent gap that grows without bound; every UCX init produces output and completes in a bounded, consistent ~11s.

The two_connected_agents fixture used by test_empty_notif_tag constructs two such agents. The first createBackend("UCX", ...) call alone (~11s) blows past the 5.0s timeout, so pytest-timeout kills the fixture setup. This is legitimately-slow work under an overly-aggressive per-test timeout, not a runaway hang — the operation was making steady progress with the same timing seen ~18 times before.

Summary: The Jenkins nixl-ci-gpu #2586 Python test stage failed: test/python/test_nixl_api.py::test_empty_notif_tag errored during fixture setup with Failed: Timeout (>5.0s) from pytest-timeout while creating a UCX backend.

Root cause: The two_connected_agents fixture constructs two nixl_agent instances, each of which does a UCX backend init. Every UCX backend instantiation in this run took a consistent ~11 seconds (seen ~18 times: 13:00:42, 13:00:53, 13:01:05 … 13:04:09), so a single agent.create_backend("UCX") alone exceeds the per-test 5.0s pytest-timeout limit. There is no unbounded silent gap — the work was making steady, bounded progress (~11s each, repeatedly), so this is a genuinely-slow init tripping an overly-tight timeout, not a deadlock. The elevated UCX init cost is consistent with the environment: the run is spamming ibv_query_port_speed(mlx5_x:1/IB, port_num=1) failed: Protocol not supported warnings and Failed to set thread affinity to NUMA node 1 errors, indicating UCX is probing IB devices whose ports it can't query, slowing device discovery on every agent creation. (The wrapping Jenkins exit code 143 / "Cancelling nested steps due to timeout" at 10:07 is a downstream symptom — the outer 10-min stage timeout fired only because the enroot export continued after the pytest failure; the real failure is the pytest error above.)

Implicated commit: unknown for the timeout value itself (the test file was last touched by the telemetry commits [REDACTED:Hex High Entropy String] / d6ae1dc, unrelated). The 5.0s pytest-timeout config governing this is the operative setting; no single code commit in the retrieved history introduced the regression — the UCX per-agent init cost (device probing) is environmental.

File: test/python/test_nixl_api.py:57 (two_connected_agents fixture) — the failing setup; timeout enforced via pytest-timeout (5.0s).

Suggested fix: Because UCX backend init is legitimately ~11s in this environment (not hung), raise the per-test/fixture timeout for the agent-creation tests — e.g. add @pytest.mark.timeout(60) to test_empty_notif_tag (and the other agent-creating tests), or bump the global pytest --timeout from 5s to a value comfortably above 2× UCX init (~30s). Separately, to attack the slowness itself, constrain UCX device discovery in CI (e.g. set UCX_NET_DEVICES / NIXL_UCX_DEVICES to a working device, or exclude the mlx5 ports returning Protocol not supported) so each agent init doesn't pay the full IB-probe cost. Note the 5.0s limit is only borderline-survivable for single-agent tests and guaranteed to fail for the two-agent fixture — that asymmetry is why earlier tests passed and this one errored.

Related: #1869 (Switch CI base image to pytorch + CUDA 13.3) and #1889 (ci: version the dispatcher pipeline) are the most relevant open CI PRs; none directly track this timeout. No existing issue found for the UCX-init-vs-pytest-timeout mismatch.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 70e7d8a2-f4b9-4372-a596-40db678649bd in the triage console for the audit trail.

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR removes the unconditional torch import in the Python API, adding a sys.modules-based helper for tensor type checks. It also introduces a shared nixl_meta_utils module for CUDA major version detection, used by nixl and nixl_ep backend selection, with corresponding build and packaging updates.

Changes

Lazy torch tensor detection in Python API

Layer / File(s)	Summary
Lazy torch tensor type resolver and usage src/api/python/_api.py	Removes module-level import torch, gates torch typing under TYPE_CHECKING, adds _is_torch_tensor() that checks sys.modules before isinstance checks, and updates get_xfer_descs/get_reg_descs to use it for single and list tensor inputs.

Shared CUDA major detection for NIXL meta packages

Layer / File(s)	Summary
detect_cuda_major implementation src/bindings/python/nixl-meta/nixl_meta_utils.py	New module implements detect_cuda_major() checking already-imported torch/cuda.bindings/cupy, then disk-based torch version read, then final torch import fallback; adds _torch_cuda_version_from_disk() to load torch/version.py without executing torch/__init__.py.
nixl backend selection src/bindings/python/nixl-meta/nixl/__init__.py	_load_cuda_backend() uses detect_cuda_major() instead of a removed torch-based helper; updates ImportError message and fallback comment.
nixl_ep backend selection src/bindings/python/nixl-meta/nixl_ep/__init__.py	_load_ep_module() uses detect_cuda_major() instead of a removed torch-based helper; updates ImportError message and fallback comment.
Build and packaging updates src/bindings/python/nixl-meta/meson.build, src/bindings/python/nixl-meta/pyproject.toml.in	meson.build copies nixl_meta_utils.py into the build directory and adds it as a custom_target input; pyproject.toml.in adds nixl_meta_utils to py-modules.

Estimated code review effort: 2 (Simple) | ~12 minutes

Suggested reviewers: itayalroy, ebarilanM, roiedanino

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: reducing NIXL import latency in Python.
Description check	✅ Passed	The description includes What and How details and is mostly complete, though a dedicated Why section is missing.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[ovidiusm]

/build

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/bindings/python/nixl-meta/pyproject.toml.in (1)

1-1: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Update copyright year range for this modified file.

This file is modified in this PR (line 40 added) but the header still reads only 2025, while the sibling meson.build in the same PR already uses 2025-2026.

📝 Proposed fix

-# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Based on learnings, for modified existing build-related files, use a year range from the earliest year to the current year, e.g. "Copyright (c) 2025-2026".

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/bindings/python/nixl-meta/pyproject.toml.in` at line 1, The SPDX
copyright header in this modified build-related file still uses only a single
year, so update the existing copyright notice to the appropriate year range for
a changed file. Use the same convention as the sibling build file and adjust the
header in pyproject.toml.in so it reflects the earliest year through the current
year, keeping the change limited to the copyright line.

Source: Learnings

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/bindings/python/nixl-meta/pyproject.toml.in`:
- Line 1: The SPDX copyright header in this modified build-related file still
uses only a single year, so update the existing copyright notice to the
appropriate year range for a changed file. Use the same convention as the
sibling build file and adjust the header in pyproject.toml.in so it reflects the
earliest year through the current year, keeping the change limited to the
copyright line.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 0f8c0d9f-d235-408d-9e78-b0f7f623c594

📥 Commits

Reviewing files that changed from the base of the PR and between 9f85e65 and ab7074d.

📒 Files selected for processing (5)

• src/bindings/python/nixl-meta/meson.build
• src/bindings/python/nixl-meta/nixl/__init__.py
• src/bindings/python/nixl-meta/nixl_ep/__init__.py
• src/bindings/python/nixl-meta/nixl_meta_utils.py
• src/bindings/python/nixl-meta/pyproject.toml.in

💤 Files with no reviewable changes (1)

• src/bindings/python/nixl-meta/nixl_meta_utils.py

[svc-nixl]

👀 Investigating nixl-ci-gpu build 2612…

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit d68ec89a

I now have enough to make a confident diagnosis. The IPC / local-transfer path (use_ipc, ipc_infos, has_ipc, uccl_engine_update_ipc_info, uccl_engine_get_ipc_info) was introduced by commit [REDACTED:Hex High Entropy String] "Enable local xfer for UCCL backend (#1428)" — this is exactly the code exercised on a single-node (mizu01) run where initiator and target are the same host, making is_local_conn == true and, for VRAM buffers, has_ipc == true.

The evidence:

• The crash is exit 139 = SIGSEGV, on the UCCL WRITE VRAM→VRAM --check_consistency job.
• Every UCCL transfer that involved VRAM ran at broken bandwidth (~0.02–0.037 GB/s, ~500µs latency, P99 Tx up to 3122µs) vs healthy UCX (2–5 GB/s). DRAM→DRAM UCCL ran at healthy ~2.3 GB/s. So the fault is specific to the VRAM IPC path.
• The consistency check reads back the destination buffer after the transfer; because the GPU-IPC local write path produced garbage / didn't map the buffer correctly, the readback dereferences a bad mapping and segfaults.

This is a noReconnectAfter/wall-time-unrelated failure — it's a genuine crash (SIGSEGV), not a hang or timeout. There are no large gaps in the log timestamps before the failure.

Summary: The nixl-ci-gpu nixlbench UCCL benchmark crashed with SIGSEGV (exit code 139) on the --backend UCCL --op_type WRITE --initiator_seg_type VRAM --target_seg_type VRAM --check_consistency case.

Root cause: The UCCL backend's local (same-node) GPU-IPC transfer path is broken for VRAM buffers. On this single-node run both nixlbench workers are on host mizu01, so uccl_engine_conn_is_local() returns true and, for VRAM (has_ipc == true), prepXfer/postXfer take the IPC branch (uccl_engine_update_ipc_info / use_ipc, ipc_ptrs passed into uccl_engine_write_vector). This path does not actually move the data correctly — visible in the log as every VRAM UCCL transfer reporting ~0.02–0.037 GB/s (vs 2–5 GB/s for UCX and ~2.3 GB/s for UCCL DRAM→DRAM). With --check_consistency the harness then reads back the destination VRAM buffer via a bad/unmapped IPC handle, dereferences invalid memory, and segfaults. The DRAM UCCL and all UCX cases pass, confirming the fault is confined to the VRAM IPC branch introduced in the "Enable local xfer for UCCL backend" change.

Implicated commit: [REDACTED:Hex High Entropy String] "Enable local xfer for UCCL backend (#1428)" — Pravein Govindan Kannan (introduced the has_ipc/ipc_info/local-xfer path). (PR #1895 under test appears to have exposed/re-enabled this UCCL VRAM+consistency case in CI rather than authoring the buggy path.)

File: src/plugins/uccl/uccl_backend.cpp:556-566 (the is_local_conn / rmd->has_ipc IPC branch in prepXfer) and :645-663 (use_ipc / ipc_ptrs in postXfer).

Suggested fix: Fix the UCCL GPU-IPC local VRAM path so the write/read vector operations actually populate and honor a valid IPC mapping (verify uccl_engine_get_ipc_info/uccl_engine_update_ipc_info produce a usable handle and that uccl_engine_write_vector uses it). As an immediate CI unblock, gate/skip the UCCL VRAM local-IPC path (or the --check_consistency VRAM UCCL cases in the nixlbench CI script) until the IPC data path is corrected — and add a completion/error check on the transfer status so a failed local IPC transfer returns NIXL_ERR_BACKEND instead of proceeding to a consistency read that segfaults.

Related: PR #1428 ([REDACTED:Hex High Entropy String], local xfer for UCCL); PR #1151 (Fix UCCL's consistency checks); PR #1595 (unified get param). No open issue found specifically tracking this segfault.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 839495b4-7050-456c-968a-35af2492a7d2 in the triage console for the audit trail.

[svc-nixl]

👀 Investigating nixl-ci-gpu build 2615…

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit d68ec89a

The Dockerfile is the container the build_helper stage builds and then exports to squashfs. Now the key question: is this a build/test hang or a genuine failure? Let me re-examine the fetched log's timestamps carefully.

The log I have is entirely the enroot squashfs export ("Failed to read file … creating empty file" at 10:08:06.6xx–10:08:06.8xx), then the mksquashfs summary at 10:09:14, and finally pipeline teardown. This export step is the crash-debug artifact from PR #1800 that runs after a failure. The actual failing command's output (the test/build that returned exit code 1) is not present in the portion of the log I retrieved — it was truncated. All I can see is the generic hudson.AbortException: script returned exit code 1 from the parallel build_helper/x86_64/ucx-v1.22.x/2 branch.

Regarding timestamps: there is no >2-minute silent gap in the fetched window — the "Failed to read file" lines are continuous milliseconds apart, then a ~68s mksquashfs (normal for a 6.8 GB image). So within the visible window there is no hang; but the visible window does not contain the root-cause command.

Per the instructions, I must not speculate about a root cause from PR titles/source alone when the actual error line was not captured. I did successfully fetch a log, but it does not contain the failing error signature. Let me be precise and honest: I have a log, but the specific failing test/build line is absent from it. I'll report what is evidenced and flag that the causal error line wasn't captured, rather than fabricate a diagnosis.

Summary: The build_helper/x86_64/ucx-v1.22.x parallel branch of nixl-ci-gpu #2615 failed with script returned exit code 1; the only substantive output captured is the post-failure enroot→squashfs container export (the "export container on test failure" crash-debug step from PR #1800), which itself completed successfully ("Export complete: /enroot_images/nixl-ci-v1.22.x-2615.sqsh").

Root cause: Not determinable from the captured log. The log window I retrieved contains only the crash-debug container-export phase (hundreds of non-fatal "Failed to read file … creating empty file" warnings from mksquashfs packing torch/torchao/triton dist-packages — these are harmless) followed by pipeline teardown. The actual command that returned exit code 1 (the build or test that triggered the export) is upstream of this window and was not included in the fetched output. The "Failed to read file" lines are a symptom of the export tool, not the failure, and there is no >2-minute stall in the captured window (the ~68s mksquashfs is normal for a 6.8 GB image), so this is a genuine non-zero exit, not a wall-clock hang.

Implicated commit: unknown (cannot attribute from captured evidence; candidates on this branch include #1868 UCX v1.22.x bump, #1869 base-image switch to pytorch+CUDA 13.3, and #1896 install python bindings from source — but none is confirmed by the log).

File: unknown (failing step not present in captured log; the exported container is built from contrib/Dockerfile).

Suggested fix: Retrieve the build_helper branch's earlier log (the test/compile step that produced exit code 1, before the crash-debug export at 10:08:06) — e.g. fetch the full nixl-ci-gpu #2615 console for the build_helper/x86_64/ucx-v1.22.x/2 branch or the corresponding test-report artifact — and re-triage on that error signature. The squashfs "creating empty file" warnings should be ignored; they are expected mksquashfs behavior and not the failure.

Related: PR #1800 (ci: export container on test failure for crash debugging) explains the export seen in the log; PR #1868 (UCX v1.22.x bump, matching the branch name) is the most likely area to inspect once the real error line is obtained.

[svc-nixl]

👀 Investigating nixl-ci-gpu build 2619…

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit d68ec89a

I now have a clear picture. Here's my diagnosis.

Diagnosis

Summary: The nixl-ci-gpu #2619 Python test suite failed: test/python/test_nixl_api.py::test_empty_notif_tag errored with Failed: Timeout (>5.0s) from pytest-timeout while its two_connected_agents fixture was creating a UCX backend (self.agent.createBackend("UCX", {}) at nixl_cu13/_api.py:401). This exited the ucx-v1.22.x branch's pytest -s test/python with code 1. The ucx-master branch's later exit code 143 at 12:41 is a downstream symptom — that branch finished all nixlbench work at 12:01:58 and then sat idle for ~40 min waiting on the already-failed pipeline before Jenkins cancelled it.

Root cause: A single UCX createBackend call hung past the 5-second pytest-timeout limit. 23 tests in the same file passed first (each successfully creating UCX backends), so this is not a universal build/link break — it is a one-off stall in UCX backend initialization during test-fixture setup. The triggering change under test is PR #1895 "[Python] Reduce latency of NIXL import", which alters the Python import/initialization path; if it changed when/how the native library and plugins are loaded (e.g. lazy loading now colliding with first backend creation), it can introduce a race/stall on the first-in-a-test UCX createBackend. The 5.0s per-test timeout is aggressive enough that a transient UCX init delay (plugin dlopen / device discovery) crosses it.

Implicated commit: PR #1895 (branch head [REDACTED:Hex High Entropy String]) — "[Python] Reduce latency of NIXL import". The test file test/python/test_nixl_api.py itself is unchanged (last touched 2026-06-16), so the regression is in the PR's import/init changes, not the test.

File: test/python/test_nixl_api.py:58 (fixture two_connected_agents) → hangs in nixl/_api.py create_backend → agent.createBackend("UCX", {}) (installed as _api.py:401).

Suggested fix:

1. Confirm reproducibility by re-running pytest -s test/python::test_empty_notif_tag on this commit vs. main. If PR [Python] Reduce latency of NIXL import #1895 deferred native-library/plugin loading, ensure the UCX plugin dlopen and device discovery are completed eagerly (or warmed) before the first createBackend, so the first backend creation isn't paying the full lazy-init cost inside a 5s-timeouted test.
2. As the correct code fix, make the import-latency optimization not defer work onto the first createBackend call; alternatively guard the lazy path so backend creation is idempotent/fast.
3. Do not simply raise the pytest timeout — the 5s limit merely exposed the stall; the real work is ensuring UCX init is deterministic and fast on first use. (Only bump the timeout if you can show UCX init is legitimately, consistently ~5s and cannot be warmed.)

Related: PR #1895 (#1895) — the PR under test. Also worth cross-checking recent RW-lock/race changes in nixl_agent.cpp (#1811 "Fixed race conditions with RW lock", #1865) if the stall proves to be a lock contention rather than a lazy-load issue.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id d472ae3b-4317-433e-8643-e3ecb9764b2f in the triage console for the audit trail.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/bindings/python/nixl-meta/nixl_meta_utils.py`:
- Around line 23-97: Add unit tests for detect_cuda_major() that exercise each
staged lookup path driven by sys.modules: imported torch, imported
cuda.bindings, imported cupy, torch-on-disk via _torch_cuda_version_from_disk(),
and the last-resort torch import. Include a torch stub where version.cuda is
None to verify it correctly falls through to the later branches instead of
stopping early. Keep the tests focused on detect_cuda_major and
_torch_cuda_version_from_disk so regressions in the fallback ordering are
caught.
- Around line 58-97: The CUDA version detection in detect_cuda_major() still
falls back to importing torch on the common CPU-only path because
_torch_cuda_version_from_disk() returns None for both “no torch” and “CPU-only
wheel”; update _torch_cuda_version_from_disk() to distinguish those cases with a
sentinel so CPU-only installs stop early, and broaden the torch.version.cuda
fallback in detect_cuda_major() to handle non-ImportError import failures by
returning None.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 23600054-7d32-4683-bb7f-bc901e392d7b

📥 Commits

Reviewing files that changed from the base of the PR and between 19e3910 and 5b9f7cb.

📒 Files selected for processing (6)

• src/api/python/_api.py
• src/bindings/python/nixl-meta/meson.build
• src/bindings/python/nixl-meta/nixl/__init__.py
• src/bindings/python/nixl-meta/nixl_ep/__init__.py
• src/bindings/python/nixl-meta/nixl_meta_utils.py
• src/bindings/python/nixl-meta/pyproject.toml.in

[ovidiusm]

/build

[svc-nixl]

👀 Investigating nixl-ci-gpu build 2624…

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit 092f23df

This is very informative. The successful master run (stage 356) shows the identical UCCL test sequence completing normally — including the first UCCL READ VRAM->VRAM case (at 14:32:44), which printed "Engine destroyed" for both processes and then cleanly proceeded to the next test. The same "Error/HUP on connection" WARN appears there too, but it's benign — both processes exited.

In the failing v1.22.x run (stage 318), the very first UCCL READ VRAM->VRAM case reached the same point (both "Engine destroyed", the epoll WARN at 14:30:25.594) but then one of the two nixlbench processes never exited. parallel --halt now,fail=1 waited for both to finish, and one hung indefinitely — 49 minutes of silence until Jenkins SIGTERM'd it at 15:19:05.

Key differences between the runs:

• The failing run uses UCX v1.22.x (stage 318); the passing run uses UCX master (stage 356). Same NIXL image otherwise.
• In stage 356 the teardown consistently logs "Stop background accept..." / "Server closed connection" then both engines exit. In stage 318's hung case, one process printed "Engine destroyed" but the peer's accept/cleanup thread never returned.

This is a nondeterministic shutdown-race/hang in the UCCL backend engine teardown, sensitive to the UCX version. It is unrelated to PR #1895 ("[Python] Reduce latency of NIXL import").

Summary: The "Run Nixlbench tests" (UCX v1.22.x) stage was aborted after the first UCCL READ VRAM→VRAM benchmark hung during engine teardown; Jenkins SIGTERM'd it (exit 143) after ~49 min of no output.

Root cause: A hang (not a timeout) in the UCCL backend's engine shutdown path. In stage 318 the benchmark result printed at 14:30:24, both workers logged "Engine destroyed" and a [WARN … event_loop rdma/epoll_client.h:207] Error/HUP on connection at 14:30:25.594 — then complete silence until the kill at 15:19:05 (~49-minute gap, the entire remaining runtime). One of the two nixlbench processes never exited after engine destruction, so parallel --halt now,fail=1 blocked forever. The identical test in the UCX-master stage (356) completed cleanly (it additionally logs "Stop background accept…"/"Server closed connection" before both engines exit), so this is a nondeterministic teardown race in the UCCL RDMA epoll event loop / background-accept thread, exposed only with UCX v1.22.x. It is unrelated to PR #1895 (a Python import-latency change), which touches no UCCL/nixlbench code.

Implicated commit: unknown (not introduced by this PR). Most relevant prior UCCL work: [REDACTED:Hex High Entropy String] (Pravein Govindan Kannan, "UCCL: Simplify and Optimize for batch transfers") and [REDACTED:Hex High Entropy String] ("Enable local xfer for UCCL backend"). The hang is in the UCCL epoll client / accept-thread shutdown.

File: UCCL backend RDMA event loop — rdma/epoll_client.h:207 (the event_loop "Error/HUP on connection" path) and the background-accept thread ("Waiting to accept incoming connection…" / "Stop background accept…") in the UCCL plugin (src/plugins/uccl).

Suggested fix:

• Re-run the v1.22.x pipeline to confirm the hang is intermittent (the master stage with the same tests passed), since it is not caused by this PR.
• In the UCCL backend teardown, make engine destruction deterministic: ensure the background-accept thread is signaled to stop and join()ed, and that the epoll event loop exits cleanly on the peer's Error/HUP, before the process returns. The missing "Stop background accept…" line in the hung case suggests the accept thread was still blocked on accept() when the peer disconnected.
• As a CI safety net, wrap each nixlbench invocation in test_nixlbench.sh with a timeout so a single hung process fails fast (with a core/stack dump) instead of consuming the whole build's wall time. Do not simply raise the Jenkins time limit — this is a hang, not slow-but-progressing work.

Related: PR #1895 (unrelated to failure); UCCL history: #1271, #1428, #895.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 90836b85-733c-40a2-8f7d-4d45d70bcbef in the triage console for the audit trail.