[yafshar]

What?

Level Zero device memory may have to be allocated with external DMA-BUF export enabled. The ze_copy memory domain advertised ze-host and ze-device in dmabuf_mem_types and returned a file descriptor from zeMemGetAllocProperties(), but allocated without an export descriptor.

• Request DMA-BUF exportability in uct_ze_copy_mem_alloc(), and only for the memory types advertised in dmabuf_mem_types, so allocating a type without a DMA-BUF path cannot fail because of the descriptor.
• Add UCX_ZE_COPY_DMABUF, following the cuda_copy and rocm_copy pattern: query export support once at md_open() with zeDeviceGetExternalMemoryProperties(), fail md_open() when y is requested but unsupported, and advertise dmabuf_mem_types and return a DMA-BUF file descriptor only when effectively enabled.
• Remove the Intel Xe module heuristic from the IB memory domain, which claimed plain ibv_reg_mr() could register ze-device memory.
• Apply the packet-tracer SGE bound whenever the IB memory domain supports DMA-BUF registration.

Why?

A DMA-BUF obtained without an export descriptor does not necessarily describe the requested range. Registration succeeds, the transfer completes, the completion is UCS_OK, and the payload is wrong. Nothing reports an error at
any layer:

ucx_perftest -t tag_lat -m ze-device -V -s 4194304   # UCX_TLS=rc,ze_copy
ucp_tests.h:166  UCX  ERROR data validation failed at offset 0: got 0x0 expected 0x1

On the tested systems, writes into ze-device memory were unaffected; failures occurred only in flows where the RDMA device read from ze-device memory. Such transfers returned UCS_OK, but the destination contained incorrect data.

How?

UCP already routes DMA-BUF registration by combining a provider's dmabuf_mem_types with an RDMA memory domain's UCT_MD_FLAG_REG_DMABUF. Therefore, removing the Xe heuristic does not disable the DMA-BUF GPUDirect RDMA path when both capabilities are available. With UCX_ZE_COPY_DMABUF=n that heuristic made UCP attempt an
ordinary registration which failed with EFAULT instead of staging through ze_copy. Removing ze-device memory from the IB MD's reg_mem_types also disabled the packet-tracer bound, hence extending it to any DMA-BUF-capable IB memory domain; otherwise tracing may dereference device memory from the CPU.

Memory allocated by an application without requesting external export cannot be repaired by UCX, so UCX_ZE_COPY_DMABUF=n is available to disable DMA-BUF registration for it.

With the default try, UCX continues to use DMA-BUF for application allocations as on master, so RDMA reads from application-allocated Level Zero device memory for which DMA-BUF export was not requested can still return incorrect data; such applications should set UCX_ZE_COPY_DMABUF=n. Level Zero specifies fd export only when DMA-BUF export was requested at allocation time. UCX_ZE_COPY_DMABUF=n disables that
path; the tested ze-device rendezvous transfers then stage through host memory. UCX_ZE_COPY_DMABUF=n can substantially increase latency for small ze-device messages because transfers stage through host memory.

test_ucp_mem_type_rndv_zcopy sends from memory allocated by ucp_mem_map(UCP_MEM_MAP_ALLOCATE) to a host buffer using a forced rendezvous zero-copy lane and verifies a non-zero pattern. It asserts that the source memory handle is registered on a memory domain used by a rendezvous zero-copy lane and that rndv/get/zcopy or rndv/put/zcopy ran on the source memory, preventing a staged-only transfer from passing silently, and skips when that lane cannot register the memory type. It is parameterized over all memory types and instantiated for rc_x, rc_x,ze_copy and rc_v,ze_copy, so CUDA and ROCm are covered where present, ZE variants skip cleanly where ze_copy is unavailable, and the ZE cases also run in builds without mlx5.

Both rendezvous suites check the protocols which moved the data as well as the payload: a forced zero-copy scheme can fall back to another transfer path (eager or AM rendezvous), so payload validation alone does not establish which protocol ran. They count protocol selections with PROTO_INFO=used on both workers around the transfer, because the tag/rndv description only estimates the rendezvous protocol which the peers then select. Their lane checks exclude memory-type-detection domains such as ze_copy, which can register their own memory types without DMA-BUF.

test_ucp_mem_type_rndv_staged runs with UCX_ZE_COPY_DMABUF=n, pins RNDV_FRAG_MEM_TYPES=host, and leaves rendezvous scheme selection to UCP. It is instantiated only for ze-host, ze-device and ze-managed, because UCX_ZE_COPY_DMABUF changes how only Level Zero memory is reached. Its rc_x,ze_copy and rc_v,ze_copy transport lists carry no CUDA or ROCm copy transport, so CUDA memory which the IB lane cannot register has no staging path there and protocol selection falls back to the reconfig stub; this failed the cuda-managed variant in CI before the suite was scoped. It still skips if a remaining RMA lane can register the source type. Otherwise it requires that no zero-copy rendezvous protocol ran on the source memory and, for ze-device, that a protocol which ran reports "frag host". On the tested systems, all three memory types ran rndv/send/ppln through host fragments.

The suite covers UCP-allocated and app-owned buffers. The latter use plain Level Zero allocations (including zeMemAllocDevice() for ze-device) and are sent without a UCP memory handle. Both suites have rc_x,ze_copy and
rc_v,ze_copy instantiations, so the ZE cases also run without mlx5.

Tested on Intel Arc Pro B60 and B70, rc over RoCE, server and client on different devices, and on an H200 NVL for CUDA. Test vehicle: this branch at 65561c7 (aff1b95, which merges 9b5c282 with master f21cfca, plus the test fixes on top), built both with and without mlx5 support on the Intel systems.

• gtest: test_ucp_mem_type_rndv_zcopy passes on rc_x and rc_v (ze-managed skips: not registrable on the tested rendezvous lanes). test_ucp_mem_type_rndv_staged passes for ze-host, ze-device and ze-managed, UCP-allocated and application-allocated, with rc_x,ze_copy and rc_v,ze_copy. On the H200 NVL, test_ucp_mem_type_rndv_zcopy passes for host and cuda and skips cuda-managed, and the staged suite has no variants, as intended. Each suite ran 3 times. The neighbouring test_ucp_mem_type, ze_copy, ze_ipc and test_md suites also ran. 0 failures in all builds.
• ucx_perftest tag_bw/ucp_put_bw/ucp_get at 8 B, 64 KiB and 4 MiB, plus -V pingpong validation (tag_lat, ucp_put_lat) at the same sizes: all pass with DMABUF=try and with DMABUF=n, on both systems and in both builds. Protocols which ran (UCX_PROTO_INFO=used) for 64 KiB and 4 MiB tagged sends between ze-device buffers: with DMABUF=try, the receiver reads the source with zero-copy RDMA, directly into the destination on the B60 and on one of the two B70 systems (zero-copy read from remote), and through host fragments copied by ze_copy on the other B70 system (read from remote, ze_copy, frag host); with DMABUF=n, the sender stages the source through host fragments with ze_copy on both (ze_copy, fenced write to remote, frag host, pipelined above 512 KiB).
• Without mlx5, the perftest RMA cases log flush(remote) is not supported from rc_verbs when UCP issues a strong fence. Master logs the same for ucp_put_bw and ucp_get; with DMABUF=n, ucp_put_lat also reaches it. All affected cases exited 0; the -V ping-pong cases validated.
• Before UCT/ZE/COPY: Initialize memtype cache when opening MD #12069, the same DMABUF=n perftest matrix passed 0 of 15 cases. A ze-device allocation made before the memtype cache was initialized could be treated as host memory, causing the ucp_mm.c memory-type assertion or a segfault in the host eager-short path. With DMA-BUF enabled, uct_ze_copy_md_mem_query() populates the cache and masks this.

Known limitation:
UCT-level ucx_perftest has no DMA-BUF registration path: uct_perf_md_mem_reg() does not supply a DMA-BUF fd. Before this change, -x rc_mlx5 -m ze-device relied on the Xe module heuristic and attempted ibv_reg_mr() on a device pointer; on the tested systems, this failed with EFAULT. After removing ze-device from the IB MD's reg_mem_types,
uct_perf_test_check_md_support() rejects this invocation up front instead:

    Unsupported memory type ze-device by rc_mlx5/...

This does not imply that ordinary registration is unsupported on every platform; only that the presence of the Xe module is insufficient evidence of that capability. Adding DMA-BUF registration to UCT-level perftest is left for a separate change.

UCP-level ucx_perftest is not subject to this UCT-level registration limitation. Its ZE allocator has no UCP-level mem_alloc callback, so UCP allocates the buffer through ucp_mem_map(UCP_MEM_MAP_ALLOCATE) and uses UCP's DMA-BUF registration path when enabled.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

test/gtest/ucp/test_ucp_mem_type.cc

The new test only exercises the default (DMABUF=try) path. Per the project test guidance, a new user-facing config that changes behavior should ideally cover both enable and disable paths. Since the whole point of this PR is that DMABUF=n must produce a correct transfer (data actually readable) while try uses the DMA-BUF handle, a variant with UCX_ZE_COPY_DMABUF=n would directly validate the regression this PR targets. Not a blocker, but worth considering.

test/gtest/ucp/test_ucp_mem_type.cc

In md_map_str(): for a remote (dst_md_index) map, the names are looked up in the local context->tl_mds[...], but dst_md_index is in the peer's MD index space, so the printed names can be wrong/misleading. It only affects the diagnostic UCS_TEST_MESSAGE, so it is cosmetic — could be clarified in the message or gated to the local case.

[yafshar]

test/gtest/ucp/test_ucp_mem_type.cc

The new test only exercises the default (DMABUF=try) path. Per the project test guidance, a new user-facing config that changes behavior should ideally cover both enable and disable paths. Since the whole point of this PR is that DMABUF=n must produce a correct transfer (data actually readable) while try uses the DMA-BUF handle, a variant with UCX_ZE_COPY_DMABUF=n would directly validate the regression this PR targets. Not a blocker, but worth considering.

test/gtest/ucp/test_ucp_mem_type.cc

In md_map_str(): for a remote (dst_md_index) map, the names are looked up in the local context->tl_mds[...], but dst_md_index is in the peer's MD index space, so the printed names can be wrong/misleading. It only affects the diagnostic UCS_TEST_MESSAGE, so it is cosmetic — could be clarified in the message or gated to the local case.

Thanks for raising both points.

md_map_str(): after tracing the endpoint direction, the existing lookup is correct. The advertised MD index originates in the remote peer's context. For GET, the map comes from rndv_zcopy_md_map(receiver(), true), whose peer is
sender(); for PUT, it contains local indices from sender(). Both therefore belong to sender()'s context, which is already passed to md_map_str().

This also makes the assertion valid: send_memh->md_map is in sender()'s index space, so its intersection with lane_md_map is meaningful because both maps use the same space. I kept the original lookup.

DMABUF=n variant: the intent is right, but on the tested systems the ze-device case in this test would skip rather than validate staging. With DMABUF=n, the IB MD is absent from reg_md_map[ZE_DEVICE], so the lane-registration check skips before transfer:

    DMABUF=try : OK    get lane mds: mlx5_1 | source memh mds: mlx5_1,ze_cpy
    DMABUF=n   : SKIP  the rendezvous zero-copy lane cannot register ze-device memory

Covering the disable path requires a separate payload-only test without the lane-registration requirement, because it validates staging rather than direct DMA-BUF access. I verified manually that ucx_perftest tag_lat -V -m ze-device
passes with UCX_ZE_COPY_DMABUF=n through staging and also passes with the default DMA-BUF path.

Focused staged-path coverage can be added separately.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

test/gtest/ucp/test_ucp_mem_type.cc

Residual coverage gap: test_ucp_mem_type_rndv_staged covers app-owned memory only with UCX_ZE_COPY_DMABUF=n, and test_ucp_mem_type_rndv_zcopy covers only UCP-allocated memory. There is no case for app-allocated ZE device memory with DMA-BUF enabled, which is the configuration finding 1 is about. Both ZE instantiations also require Intel GPU hardware, so CI will skip them unless a ZE runner is in the loop.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (AddressSanitizer gpu on worker 0) · commit 6c01e4a8

TL;DR: The new gtest test_ucp_mem_type_rndv_staged.staged_app_mem fails for the cuda-managed variant on the CUDA ASAN worker because the test is instantiated over all memory types while its premise (UCX_ZE_COPY_DMABUF=n ⇒ payload must be staged) only holds for Level-Zero memory. Restrict the new test's variants to ZE memory types (or skip CPU-accessible/non-ZE memtypes).

Full analysis

Summary: gtest run in the "AddressSanitizer gpu on worker 0" job ended with 1 FAILED TEST: rcv_ze/test_ucp_mem_type_rndv_staged.staged_app_mem/2, where GetParam() = rc_v,ze_copy/cuda-managed, causing make: *** [Makefile:4925: test] Error 1 and Bash exited with code '2'. 10339/10340 tests passed; no sanitizer report, no hang (log shows continuous activity to the end).

Root cause: test_ucp_mem_type_rndv_staged is new in this PR and inherits test_ucp_mem_type's variant generator, which uses add_variant_memtypes() — so it is instantiated for every memory type (host, cuda, cuda-managed, rocm…), not only Level Zero. The class only sets UCX_ZE_COPY_DMABUF=n (test_ucp_mem_type.cc:430), which has no effect on CUDA memory. On this CI worker there is no Intel GPU, so the ze_copy transport contributes nothing and the cuda-managed variant is exercised against the rc_v lane.

For cuda-managed the test's gate can_reg = sender().ucph()->reg_md_map[send_mem_type] & lane_md_map (line 319) evaluates false, so the XFER_STAGED skip at line 324-328 does not trigger and the test proceeds. But cuda-managed memory is CPU-accessible, so UCP protocol selection can legitimately pick a zero-copy rendezvous (the IB MD can register the managed pages as host memory even though the reg_md_map[cuda-managed] bit is unset). That makes the assertion at lines 340-341 — EXPECT_EQ(std::string::npos, proto_desc.find("zero-copy")) << "the source memory is accessed by the transport" — fail. In short, the reg_md_map bit used as the "cannot register" oracle disagrees with actual protocol selection for CPU-accessible managed memory, and that variant should never have been generated for a ZE-DMA-BUF-specific test.

Implicated commit: [REDACTED:Hex High Entropy String] — "UCT/ZE/COPY: Test transfers with DMA-BUF disabled", Yaser Afshar (the PR head commit; the test file was previously touched only by 53c855b by the same author)

File: test/gtest/ucp/test_ucp_mem_type.cc:426-449 (class/instantiation), with the failing assertion at test/gtest/ucp/test_ucp_mem_type.cc:340 and the faulty gate at :319-328

Suggested fix: Scope the new tests to Level Zero memory instead of inheriting the all-memtypes variant set. Concretely, in test_ucp_mem_type_rndv_staged (and test_ucp_mem_type_rndv_zcopy, which has the same exposure), override get_test_variants() to add only ZE memory types, e.g.:

static void get_test_variants(std::vector<ucp_test_variant> &variants) {
    add_variant_values(variants, get_test_variants_base,
                       UCS_MEMORY_TYPE_ZE_DEVICE, "ze-device");
    add_variant_values(variants, get_test_variants_base,
                       UCS_MEMORY_TYPE_ZE_HOST, "ze-host");
}

As a minimal alternative, add an early skip in test_xfer_from_mem_type() when the tested memory type is not a ZE type (UCS_TEST_SKIP_R("DMA-BUF export control applies to Level Zero memory only")), and additionally skip XFER_STAGED for memory types in UCS_MEMORY_TYPES_CPU_ACCESSIBLE, since for those the reg_md_map-based can_reg check cannot predict whether protocol selection picks zero-copy. Re-run the ASAN GPU job after the change to confirm the cuda-managed variant is no longer generated.

Related: PR #11903 "UCT/ZE/COPY: Allocate DMA-BUF-exportable memory for RDMA" (#11903); no pre-existing issue found for this test name.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 8916502f-a4a7-4655-b091-7efd196cbe44 in the triage console for the audit trail.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[yafshar]

test/gtest/ucp/test_ucp_mem_type.cc

Residual coverage gap: test_ucp_mem_type_rndv_staged covers app-owned memory only with UCX_ZE_COPY_DMABUF=n, and test_ucp_mem_type_rndv_zcopy covers only UCP-allocated memory. There is no case for app-allocated ZE device memory with DMA-BUF enabled, which is the configuration finding 1 is about. Both ZE instantiations also require Intel GPU hardware, so CI will skip them unless a ZE runner is in the loop.

The application-allocated + DMA-BUF-enabled case was exercised manually: zero-copy is selected and the payload validates on the tested system. It is not added as a permanent test, since it would assert zero-copy on memory whose fd Level Zero does not guarantee; see the thread on ze_copy_md.c:39. The ZE instantiations skip without an Intel GPU, like the CUDA ones; a ZE runner is a separate effort.

[svc-nvidia-pr-review]

src/uct/ze/copy/ze_copy_md.c

Residual coverage gap: there is no test for the zeMemAlloc* failure path described above (allocation refused because of the export descriptor).

Residual coverage gap: both ZE instantiations (rcx_ze, rcv_ze) need Intel GPU hardware plus an RDMA NIC; CI will skip them unless a ZE runner is in the loop, so the DMA-BUF correctness claim rests on the author's local run.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 0) · commit ac618d10

TL;DR: The AddressSanitizer "roce on worker 0" job was not a test or compile failure — the Azure Pipelines agent itself was shut down mid-build ("The agent has received a shutdown signal"), cancelling the job while gtest sources were still compiling. Just re-run the pipeline; no code change is warranted.

Full analysis

Summary: Build 138647, job "roce on worker 0" (./contrib/test_jenkins.sh, ASan config) was cancelled during the make of test/gtest — no compiler error, no test ever ran.

Root cause: Infrastructure, not code. The log shows continuous, healthy progress with no stalls: object files are being emitted every 2–20 seconds from 22:10:21 through 22:22:37 (last compile: ucp/test_ucp_context.cc). At 22:22:33 the agent emitted:

• ##[error]The agent has received a shutdown signal. This can happen when the agent service is stopped, or a manually started agent is canceled.
• ##[error]The Operation will be canceled. The next steps may not contain expected logs.
• ##[error]The operation was canceled. → Finishing: roce on worker 0

Because the largest gap between consecutive log lines is well under a minute, this was not a hang and not a wall-clock timeout of the job's own work — the hosting agent/VM went away (agent service stop, pod eviction, or spot-node reclaim) while the ASan build was still ~2/3 of the way through compiling test/gtest. Note the ASan gtest build is the long pole here: ~12 minutes of pure compilation before any test binary is even linked, which widens the window for an agent reclaim to hit this job.

Nothing in the log implicates the PR's branch content (fix/ze-dmabuf-exportable-allocation): every gcc/g++ invocation up to the cancellation succeeded, including all src/ucp, src/uct, src/ucs libraries and the perftest/iodemo/examples targets — the ZE/dmabuf changes compiled and linked cleanly.

Implicated commit: none — [REDACTED:Hex High Entropy String] is not implicated; the failure is agent-side.

File: n/a (no source error). Cancellation point: test/gtest/ucp/test_ucp_context.cc compile, build log line ##[error]The agent has received a shutdown signal

Suggested fix: Re-run the failed job ("roce on worker 0") — this is a transient agent reclaim and should pass on retry. If this recurs on the ASan legs specifically, it's worth reducing the exposure window rather than chasing a code bug: (a) check the health/preemptibility of the agent pool backing the roce workers (spot/preemptible nodes are the usual culprit for shutdown signal), and (b) consider enabling parallel compilation (make -j) and/or ccache for the ASan gtest build in contrib/test_jenkins.sh, since ~12 minutes of serialized test/gtest compilation is what left the job sitting on the agent long enough to be reclaimed. Do not raise any time limit — the job was making steady progress and was externally killed.

Related: none — searches for the agent-shutdown signature returned no matching tracked issue; the hits (#11932, #11842, #11970) are unrelated.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 90c2f17d-4778-448e-913a-0c78bd3e7f24 in the triage console for the audit trail.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (Tests gpu on worker 0) · commit 4139ae87

TL;DR: The gtest run failed on a single unrelated test, tcp/test_uct_sockaddr.err_handle/4 (/eno1), which timed out waiting for the client to report "server unavailable" when connecting to an unused port on the external eno1 address; this is an environment-dependent flake in the TCP CM negative-connect test, not a regression from PR #11903 (which only touches src/uct/ze). Re-run the job, and harden err_handle by bounding the TCP connect timeout the same way test_uct_sockaddr_err_handle_non_exist_ip already does.

Full analysis

Summary: make test in build-test/test/gtest exited with Error 1 because 1 of 10427 tests failed: [ FAILED ] tcp/test_uct_sockaddr.err_handle/4, where GetParam() = /eno1 (10406 passed, 20 skipped).

Root cause: err_handle calls connect() to a random, non-listening port and then waits up to the default 10 s (DEFAULT_TIMEOUT_SEC, further capped by ucs::get_deadline()) for TEST_STATE_CLIENT_GOT_SERVER_UNAVAILABLE. That bit is only set when the client CM callback reports UCS_ERR_UNREACHABLE/NOT_CONNECTED/CONNECTION_RESET (test_sockaddr.cc:212-216). For the /eno1 parameterization the connect targets the host's real external IP (10.224.36.98 per the surrounding log lines) instead of loopback, so no immediate ECONNREFUSED RST is guaranteed — if the SYN is dropped/filtered (or the randomly chosen port is transiently occupied by another process), ucs_socket_connect() stays UCS_INPROGRESS and no EPOLLERR arrives before the 10 s deadline, so EXPECT_TRUE(...) at line 1003 fails. Notably, the sibling negative test test_uct_sockaddr_err_handle_non_exist_ip explicitly bounds this by setting TCP_CM_SYN_CNT=1 and TCP_CM_USER_TIMEOUT=1s (added in commit b82a920, "UCT/TCP: Bound negative connect test timeout", #11415), and also uses a 300 s wait — err_handle has neither protection. The PR under test only modifies src/uct/ze (commits ac618d1, [REDACTED:Hex High Entropy String], 53c855b, all by Yaser Afshar), and no ZE/CUDA transport is involved in a tcp CM test, so the failure is unrelated to the change.

Implicated commit: unknown (pre-existing test fragility; the related hardening commit that missed this test is b82a920, Yossi Itigin — "UCT/TCP: Bound negative connect test timeout", #11415). The PR commits under test (ac618d1 / [REDACTED:Hex High Entropy String] / 53c855b, Yaser Afshar) touch only src/uct/ze and are not implicated.

File: test/gtest/uct/test_sockaddr.cc:990-1004 (failing assertion at line 1003); timeout default at test/gtest/common/test.h:84; bounding config applied only at test/gtest/uct/test_sockaddr.cc:1140-1153

Suggested fix:

1. Short term: re-run the "gpu on worker 0" job — this is a known-class flake independent of the ZE DMA-BUF change, and the PR should not be blocked on it.
2. Proper fix: move the connect-timeout bounding out of test_uct_sockaddr_err_handle_non_exist_ip::init() into a shared base used by err_handle (and conn_to_non_exist_server_port), i.e. add to the fixture init():
   • modify_config("TCP_CM_SYN_CNT", "1", IGNORE_IF_NOT_EXIST);
   • modify_config("TCP_CM_USER_TIMEOUT", "1s", IGNORE_IF_NOT_EXIST);
     and raise the wait_for_bits(&m_state, TEST_STATE_CLIENT_GOT_SERVER_UNAVAILABLE) timeout from the implicit 10 s to an explicit larger value (as done with 300 on line 1183), so a slow-but-eventual TCP error is still observed rather than failing the test.
3. Optionally make the test deterministic: bind-and-close a socket to reserve/verify the target port is genuinely free, or target the loopback address for this negative case so an immediate RST is guaranteed.

Related: #11415 (bounded negative connect timeout for the non-existent-IP variant only); PR under test: #11903

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id f80bf9ba-3612-48c6-998f-d6c0a44405f3 in the triage console for the audit trail.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (Tests new on worker 0) · commit 5582a585

TL;DR: The Tests new on worker 0 job failed because a single gtest case — tcp/test_ucp_rma.put_nonblocking/0 <tcp,cuda_copy,rocm_copy/flush_worker> — failed out of 8901, in the TCP/RMA data path that this PR (Level-Zero ze_copy DMA-BUF allocation) does not touch; this is an unrelated/flaky failure, so re-trigger the job and, if it reproduces, chase the RMA put data-validation path rather than the ZE change.

Full analysis

Summary: make test in the gtest stage exited with code 1 (Makefile:4992: test) after [ FAILED ] tcp/test_ucp_rma.put_nonblocking/0, where GetParam() = tcp,cuda_copy,rocm_copy/flush_worker — 8900 of 8901 tests passed.

Root cause: The only failing case is a TCP RMA ucp_put_nbx non-blocking test with worker flush; the test body runs test_ucp_rma::put_nbi over host/CUDA/ROCm memory-type pairs and the only failure point after the sends is the post-flush buffer comparison (ADD_FAILURE() << "data validation failed" in test_ucp_memheap::test_xfer) or a ucp_mem_map/rkey ASSERT_UCS_OK. Nothing in this path involves Level-Zero: PR #11903's only code change is in src/uct/ze/copy (commit b402de77, Yaser Afshar, "check DMA-BUF before context, rename flag"), and the failing variant instantiates only tcp, cuda_copy and rocm_copy transports — the ZE MD is not even opened. The run also shows no timeout/hang (continuous timestamps, test suite finished its teardown normally), so this is a genuine single-test assertion, not an infrastructure kill, and it is pre-existing/intermittent rather than introduced by this branch. Note: the per-test assertion text itself is beyond the retrievable tail of the Azure log (the failure happened early in a ~3100 s run); the diagnosis above rests on the gtest summary line plus the variant parameters.

Implicated commit: none for this failure — PR head [REDACTED:Hex High Entropy String] / b402de77 (Yaser Afshar) only modifies src/uct/ze/copy, which the failing variant does not exercise.

File: test/gtest/ucp/test_ucp_rma.cc:283 (UCS_TEST_P(test_ucp_rma, put_nonblocking)), failure asserted via test/gtest/ucp/test_ucp_memheap.cc:115

Suggested fix: Re-run the Tests new on worker 0 job to confirm flakiness — no change to the PR is warranted on this evidence. To get a usable signature if it recurs, re-run the single case on the worker with the full output preserved, e.g. GTEST_FILTER='tcp/test_ucp_rma.put_nonblocking/0' UCX_LOG_LEVEL=info ./gtest and capture the [ FAILED ] block plus the preceding [ INFO ] host->cuda NxSIZE progress line; that pinpoints whether it is a data-validation mismatch (TCP RMA put completion/flush ordering) or a ucp_mem_map error. Separately, consider having the CI upload the gtest output as an artifact so the assertion text survives log truncation on 50-minute runs.

Related: #11903 (this PR); no existing issue found tracking tcp/test_ucp_rma.put_nonblocking flakiness — worth opening one if it recurs.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id c06cd712-7112-4910-a27b-5403e098481a in the triage console for the audit trail.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

src/uct/ze/copy/ze_copy_md.c

Can we add a ZE_COPY_DMABUF=n case to test/gtest/uct/test_md.cc, next to test_rocm_copy::dmabuf_disable? That one only needs an Intel GPU, while both new UCP tests need a GPU and an RDMA NIC, so the disable path would otherwise have no reachable coverage.

test/gtest/ucp/test_ucp_mem_type.cc

dmabuf_reg_md_map is the set of MDs that can consume a DMA-BUF, not the ones that did, so for cuda/rocm this passes whenever the buffer is registered on any IB MD — including via nv_peer_mem. Can we reword the message, or keep the check under dmabuf_only where it is actually conclusive?

src/uct/ib/base/ib_md.c

minor: UCT_MD_FLAG_REG_DMABUF is set on any mlx5 device with a recent kernel, so this also truncates UCX_LOG_LEVEL=data payload dumps to the first SGE on host-only runs. Worth mentioning in the comment above so it does not look like a GPU-only change.

Note (not blocking): ZE_HOST export is gated on a device-scoped zeDeviceGetExternalMemoryProperties() query, and there is no test for a zeMemAlloc* failure caused by the added export descriptor.

Note (not blocking): app-allocated ZE memory with DMABUF=try can still yield an fd that does not describe the requested range; the PR documents this in the config help rather than detecting it.

Note (not blocking): rcx_ze/rcv_ze instantiations need an Intel GPU plus an RDMA NIC, so CI will skip them without a ZE runner.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (AddressSanitizer BlueField on worker 0) · commit 4a2d4f44

TL;DR: The gtest shard didn't fail an assertion — it hung: dc_mlx5/test_p2p_rma_madvise.madvise/1 sat in uct_p2p_test::blocking_send()'s unbounded completion-polling loop for exactly 900 s until the gtest watchdog fired SIGABRT. The stuck DC get_zcopy completion on the BlueField device is unrelated to PR #11903 (which only touches src/uct/ze/copy), so re-run the job; the real actionable bug is that blocking_send() has no deadline, so a lost completion burns the whole 200-min test budget instead of reporting a diagnosable error.

Full analysis

Summary: AddressSanitizer BlueField gtest shard 1/4 aborted because dc_mlx5/test_p2p_rma_madvise.madvise/1 never completed a get_zcopy operation; the test watchdog killed it after 15 minutes (make: *** [Makefile:4992: test] Aborted (core dumped)).

Root cause: A hang, not a slow test. Log timestamps show one single gap covering the entire failure: 18:54:12.57 [ RUN ] dc_mlx5/test_p2p_rma_madvise.madvise/1 <dc_mlx5/mlx5_0:1/loopback> → nothing at all until 19:09:12.56 ... test_helpers.cc:57: Failure / Connection timed out - abort testing — exactly the 900 s watchdog_timeout in test/gtest/common/test_helpers.cc:31. The watchdog pthread_killed the test thread (signal 6), and the backtrace shows where it was parked:
uct_test::progress() (uct_test.cc:801) ← uct_p2p_test::blocking_send() (uct_p2p_test.cc:317) ← test_p2p_rma_madvise_madvise_Test::test_body() (test_p2p_rma.cc:166).
Line 166 is the first get_zcopy of the test (before the system()/fork part), and uct_p2p_test.cc:316-318 is a bare while (m_completion_count <= prev_comp_count) { progress(); } loop with no deadline — so a completion that never arrives on the dc_mlx5 loopback endpoint spins until the watchdog kills the process.

Nothing in the log points at the PR: the only transports probed are IB/mlx5 (srd/ugni unavailable, no Level-Zero MD on this BlueField), the buffers use the first access_mem_types bit (HOST), and the three branch commits (4a2d4f44, 75f974d2, b402de77 by Yaser Afshar) modify only src/uct/ze/copy, which is never loaded on this runner. This is an environment/transport-level flake (a lost DC completion on the BF HCA under ASAN), not a regression from the diff.

Implicated commit: none for the hang — branch commits (4a2d4f44, Yaser Afshar) touch only src/uct/ze and cannot affect the dc_mlx5 HOST-memory path. The exposed test-infrastructure weakness predates this PR (test/gtest/uct/uct_p2p_test.cc, unbounded wait loop).

File: test/gtest/uct/uct_p2p_test.cc:316 (unbounded completion wait), triggered from test/gtest/uct/test_p2p_rma.cc:166; watchdog at test/gtest/common/test_helpers.cc:31,57

Suggested fix:

1. Re-trigger the BlueField ASAN job — this failure is not attributable to the PR diff and should not block it.
2. Harden the test so the next occurrence is diagnosable instead of a 15-minute blackout: bound the wait in uct_p2p_test::blocking_send() with a deadline, e.g. replace the loop at uct_p2p_test.cc:316 with something like
   ucs_time_t deadline = ucs::get_deadline(); while (m_completion_count <= prev_comp_count) { progress(); ASSERT_LT(ucs_get_time(), deadline) << "get_zcopy completion timed out"; }
   and dump iface/QP state (uct_iface_print_info / UCX_LOG_LEVEL=debug) on timeout.
3. If this recurs on BlueField, investigate DC completion handling on that device — recent DC/mlx5 work worth bisecting against is f53da7f0 (refcounted TTL-based AH cache), c8495100 (AH cache TTL knob) and [REDACTED:Hex High Entropy String] (outstanding purge).

Related: PR #11903 (the PR under test, ZE-only changes); candidate DC/zcopy completion areas: #11874 ("UCT/IB/MLX5: support put zcopy in purge outstanding"), #12066 ("UCT/IB/RC/VERBS: Support QPs without SRQ")

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id afc50878-b255-4827-8f6c-ac02df56224d in the triage console for the audit trail.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[yafshar]

Thanks.

src/uct/ze/copy/ze_copy_md.c

Can we add a ZE_COPY_DMABUF=n case to test/gtest/uct/test_md.cc, next to test_rocm_copy::dmabuf_disable? That one only needs an Intel GPU, while both new UCP tests need a GPU and an RDMA NIC, so the disable path would otherwise have no reachable coverage.

ZE_COPY_DMABUF=n is covered by the UCP _off tests, which passed on B60/B70. Agreed that an MD test would additionally cover GPU-only systems and direct fd queries. I'll add that in the follow-up; this PR currently adds 496 lines.

test/gtest/ucp/test_ucp_mem_type.cc

dmabuf_reg_md_map is the set of MDs that can consume a DMA-BUF, not the ones that did, so for cuda/rocm this passes whenever the buffer is registered on any IB MD — including via nv_peer_mem. Can we reword the message, or keep the check under dmabuf_only where it is actually conclusive?

Reworded in 346c123 to "memory handle registration on a DMA-BUF capable memory domain". For CUDA and ROCm with try, the check stays as a registration check; uct_p2p_rma_test_dmabuf is the test that shows the NIC accessed the memory through the DMA-BUF.

src/uct/ib/base/ib_md.c

minor: UCT_MD_FLAG_REG_DMABUF is set on any mlx5 device with a recent kernel, so this also truncates UCX_LOG_LEVEL=data payload dumps to the first SGE on host-only runs. Worth mentioning in the comment above so it does not look like a GPU-only change.

Done in 346c123. The comment now says that the limit also applies to host-only runs on a DMA-BUF capable device.

Note (not blocking): ZE_HOST export is gated on a device-scoped zeDeviceGetExternalMemoryProperties() query, and there is no test for a zeMemAlloc* failure caused by the added export descriptor.

Note (not blocking): app-allocated ZE memory with DMABUF=try can still yield an fd that does not describe the requested range; the PR documents this in the config help rather than detecting it.

Note (not blocking): rcx_ze/rcv_ze instantiations need an Intel GPU plus an RDMA NIC, so CI will skip them without a ZE runner.

ze-host export uses the same device query as dmabuf_mem_types, so a device that does not report DMA-BUF export gets no export descriptor; there is no test for an allocation failure caused by the descriptor. The application-allocated case is the previously agreed follow-up. The instantiations are now rcx and rcv. Their ZE variants require an Intel GPU and an RDMA NIC; CUDA and ROCm variants do not require Intel hardware.

[svc-nvidia-pr-review]

Residual note: App-allocated ZE memory with DMABUF=try can still yield an fd that does not describe the requested range; documented in the config help rather than detected.

Residual note: No test for a zeMemAlloc* failure caused by the added export descriptor (notably ZE_HOST, gated on a device-scoped export-properties query).

Residual note: The memh->md_map & dmabuf_reg_md_map check in test_ucp_mem_type_dmabuf::test_xfer() is only conclusive for the dmabuf_only memtypes; for cuda/rocm it passes whenever any IB MD registered the buffer.

Residual note on coverage: the rcx/rcv instantiations exercise cuda/rocm memtypes on IB runners, but the ZE paths (and the new uct_p2p_rma_test_dmabuf ZE legs) need an Intel GPU plus an RDMA NIC, so CI will skip them without a ZE runner.

Residual note on size: ~480 added lines, under the 500 limit now; ~390 of them are tests.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (Tests new on worker 1) · commit 346c1231

TL;DR: The gtest binary aborted on an rcache assertion (!(region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE)) while ucp_mem_unmap() released the parent registration-cache region in tcp/test_ucp_rma.put_nonblocking/2 <tcp,cuda_copy,rocm_copy/user_memh>; the parent region's refcount hit 0 while it was still in the page table (a refcount under-count in the lock-free rcache put path), and nothing in that host/TCP code path touches the ZE DMA-BUF change under test — so this is an unrelated (pre-existing, intermittent) failure, not a regression from PR #11903.

Full analysis

Summary: make test aborted (core dumped) in the UCP gtest: rcache.c:405 Assertion '!(region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE)' failed during ucp_mem_unmap() of the receiver's memheap in tcp/test_ucp_rma.put_nonblocking/2 <user_memh>.

Root cause: From the backtrace in the log:
test_ucp_memheap::test_xfer() (test_ucp_memheap.cc:120, ucp_mem_unmap(receiver, memheap_memh)) → ucp_mem_unmap → ucp_memh_cleanup → ucp_memh_put_rcache → ucs_rcache_region_put_unsafe → ucs_mem_region_destroy_internal.
A cached rcache region is created with refcount = 2 ("Page-table + user", src/ucs/memory/rcache.c:1028), and the page-table reference is only dropped by ucs_rcache_region_invalidate_internal(), which also clears UCS_RCACHE_REGION_FLAG_PGTABLE (rcache.c:492-501). The assertion therefore means the user memh's single put drove the refcount of a still-in-page-table region to 0, i.e. one reference was lost (an extra put, or a lost increment) before the unmap. The UCP release path ucp_memh_put_rcache() (src/ucp/core/ucp_mm.c:441) calls the lock-free ucs_rcache_region_put_unsafe(), which mutates the refcount non-atomically (--region->refcount, src/ucs/memory/rcache.inl:127) and calls ucs_mem_region_destroy_internal() — a function documented as "Lock must be held in write mode" — while holding only the UCP context mt_lock (a no-op in single-threaded mode). The matching acquire path ucs_rcache_lookup_unsafe() also does a plain region->refcount++ under the read lock (rcache.inl:103), whereas the invalidation/eviction paths (UCM vm_unmapped callback, ucs_rcache_invalidate_handler on the async thread) use atomics. Any overlap of those paths loses one refcount update, which produces exactly this signature. The test that crashed uses host memory over TCP (tcp,cuda_copy,rocm_copy), with no ZE/DMA-BUF transport involved, so the branch's ZE allocation change is not exercised here.

Implicated commit: unknown — not attributable to 346c1231 / PR #11903. The lock-free put path that violates the "write lock held" contract originates in the older rcache locking changes 0ed92911 ("UCP/MM: Remove rcache locking") and 37c574df ("UCP/MM: lock free when deregistration memory region", Artemy Kovalyov); last functional touches to the region refcount paths are 311e3506 (nbellalou) and 5687e9a8 (Colin Hirsch).

File: src/ucs/memory/rcache.c:396 (assert; line 405 in the built tree) — reached from src/ucs/memory/rcache.inl:127-128 and src/ucp/core/ucp_mm.c:441 (ucp_memh_put_rcache), test site test/gtest/ucp/test_ucp_memheap.cc:120

Suggested fix:

1. For the PR: re-run the "Tests new on worker 1" job — this failure is independent of the ZE DMA-BUF change and should not block UCT/ZE/COPY: Allocate DMA-BUF-exportable memory for RDMA #11903.
2. For the real defect, open a tracking issue and harden the rcache refcounting: make the fast-path refcount updates atomic (ucs_atomic_add32/ucs_atomic_fsub32) in ucs_rcache_lookup_unsafe() (rcache.inl:103) and ucs_rcache_region_put_unsafe() (rcache.inl:127), and have ucp_memh_put_rcache() take rcache->pgt_lock in write mode (or route through ucs_rcache_region_put() / UCS_RCACHE_REGION_PUT_FLAG_TAKE_PGLOCK) when the put can reach ucs_mem_region_destroy_internal().
3. To make future occurrences diagnosable, upgrade the bare ucs_assert at rcache.c:396 to ucs_assertv printing region, start..end, flags, refcount and rcache->name, mirroring the assert above it.

Related: PR under test: #11903 — no existing issue found for this assertion signature; worth filing one.

[tvegas1]

/build

[svc-ucx]

🤖 CI Triage Agent — UCX-OSS/ucx-codestyle-oss · commit 346c1231

TL;DR: The Format Check stage failed because the PR's changed lines are not clang-format clean — git-clang-format produced a non-empty format.patch (ze_copy_md.c config table/ternary plus four gtest files), so the script applied it, saw a diff and exited 1. Fix: run the repo formatter over the branch's changed lines and commit the result.

Full analysis

Summary: UCX-OSS/ucx-codestyle-oss #62, stage "Format Check" (node 68) exited 1 with ERROR: Code is not formatted according to code style.

Root cause: Not a test or infra failure. The stage runs git-clang-format over the lines changed by branch fix/ze-dmabuf-exportable-allocation, writes the result to format.patch, and fails if the patch is non-empty (git apply format.patch + git diff --quiet --exit-code). The patch is non-empty, i.e. the hand-written formatting in the new/modified code disagrees with the repo .clang-format (ColumnLimit 80, ContinuationIndentWidth: 8, AlignAfterOpenBracket: true, and UCS_TEST_P / UCS_TEST_SKIP_COND_P / ucs_for_each_bit listed under ForEachMacros, so their bodies must use ) { on the same line instead of a brace on the next line). Concretely clang-format wants:

• src/uct/ze/copy/ze_copy_md.c:30-51 — the uct_ze_copy_md_config_table[] initializer (touched by the new DMABUF entry) re-wrapped as [] =\n {{...}, ..., {NULL}};
• src/uct/ze/copy/ze_copy_md.c:96-97 — ternary for md_attr->dmabuf_mem_types broken across three lines
• test/gtest/ucp/test_ucp_mem_type.cc:118-122, 177-180, 208, 221-224, 256-264 — continuation indent and UCS_TEST_P(...) { brace placement
• test/gtest/uct/ib/test_ib_xfer.cc:292-295, 320-325, 338-345, 391-394, 408-416 — argument wrapping, lambda args, UCS_TEST_SKIP_COND_P(...) {
• test/gtest/uct/test_md.cc:271-274 — && operand wrapping

Note the surrounding legacy code (e.g. src/uct/cuda/cuda_copy/cuda_copy_md.c:40-62) is itself not clang-format clean, which is why the check is diff-scoped and why touching the ZE config table dragged the whole initializer into the formatting range.

Implicated commit: [REDACTED:Hex High Entropy String] — "UCT/ZE/COPY: Address review: clarify DMA-BUF comments, help text and test message" (Yaser Afshar); earlier branch commits 4a2d4f4 / 75f974d contribute the gtest hunks.

File: src/uct/ze/copy/ze_copy_md.c:30 (also :96), test/gtest/ucp/test_ucp_mem_type.cc:208, test/gtest/uct/ib/test_ib_xfer.cc:292, test/gtest/uct/test_md.cc:271

Suggested fix: Locally run the same formatter over the branch and amend, e.g.
git-clang-format --style=file $(git merge-base HEAD upstream/master) (or the repo's codestyle helper used by buildlib/tools/codestyle.sh's format step), then git commit -a --amend/add a fixup commit. If the mechanical {{ ... }} reflow of uct_ze_copy_md_config_table is considered unreadable, wrap that table in /* clang-format off */ ... /* clang-format on */ instead of hand-formatting it — otherwise the check will keep failing. The brace-on-same-line change for UCS_TEST_P/UCS_TEST_SKIP_COND_P bodies is intended behavior (those macros are in .clang-format ForEachMacros), so accept it.

Related: PR #11903 (this branch); no existing issue found for this formatter behavior (nearest hit: PR #11923).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 29e3bb6e-ad6b-4e52-b857-cec8beb154f1 in the triage console for the audit trail.

[svc-ucx]

🤖 CI Triage Agent — UCX-OSS/ucx-build-oss · commit 346c1231

TL;DR: One of 40 parallel "Build" rows failed with ./buildlib/tools/builds.sh: No such file or directory (exit 127) because that agent/pod's workspace was empty — the source tree was never populated in that container — not because of anything in PR #11903; fix the Jenkins pipeline to checkout/verify the tree inside each build branch (and retry) rather than assuming a shared workspace.

Full analysis

Summary: Build #47 failed in the Parallel → Build branch with stage ID 875; the shell step exited 127 with script.sh.copy: line 4: ./buildlib/tools/builds.sh: No such file or directory.

Root cause: Evidence from the logs:

• Failing stage 875 log is a single line: [REDACTED:Base64 High Entropy String]@tmp/durable-1ad0da52/script.sh.copy: line 4: ./buildlib/tools/builds.sh: No such file or directory, and the build aborts with Step Build failed with exit code=127.
• The branch's timeout step (node 876) logged at 17:04:06 and the failure came at 17:05:43 — ~97 s later, so there is no multi-minute gap: nothing hung, the script simply wasn't on disk. The stage's 23-min duration is pod provisioning/queueing, not work.
• buildlib/tools/builds.sh does exist in the repo at this commit (readable, with the ZE/require_ze handling), and 39 other parallel rows ran the exact same ./buildlib/tools/builds.sh successfully in this very build (e.g. stage 1256 completed RPM packaging for el9/x86_64, stage 847 etc.). Build UCS/CLASS: Add macros to declare new/delete prototypes #46 on the same job had all 43 rows green.
• The durable-task @tmp/durable-* directory was created, so the workspace path existed but contained no checked-out source.

So this is a CI infrastructure/workspace problem on one agent pod: the per-branch source provisioning (checkout / stash-unstash / shared workspace volume) did not populate [REDACTED:Base64 High Entropy String] in that container, while the pipeline ran ./buildlib/tools/builds.sh unconditionally. It is unrelated to the ZE dmabuf change in PR #11903 (nothing in the log touches uct/ze beyond normal successful builds).

Secondary (non-fatal) noise seen in the same logs: Module dev/fuse-3.10.5 not found / /hpc/local not mounted on these pods, so build_fuse is skipped everywhere — worth fixing separately so FUSE coverage isn't silently lost.

Implicated commit: none in the product code — infrastructure. Closest CI-related changes: 9af60a8 (Alexey Rivkin, "CI: Add Fedora 44, Ubuntu 26.10, Azure Linux 4.0, UOS and openEuler 22.03 SP4 / 24.03 SP3", #11906) added new distro build rows, and the Jenkins pipeline itself is still being introduced (#12094 / #12166). The Jenkinsfile/pod templates are not in this repo, so the exact row cannot be pinned from the logs.

File: Jenkins pipeline definition for UCX-OSS/ucx-build-oss (not in repo; the invoked script is buildlib/tools/builds.sh:1)

Suggested fix:

1. In each parallel Build branch, make source availability explicit and verified before invoking the script, e.g.:

   container(img) {
     checkout scm            // or unstash 'src'
     sh 'test -f buildlib/tools/builds.sh || { ls -la; echo "workspace not populated"; exit 1; }'
     sh './buildlib/tools/builds.sh'
   }

   so a missing workspace fails fast with a diagnosable message instead of a bare exit 127.
2. Wrap the checkout/unstash in retry(3) (the pod is already inside a retry/podTemplate), so a transient workspace-provisioning miss self-heals instead of failing the whole 40-row matrix.
3. Verify the pod template for the affected row mounts the workspace volume and sets the same workingDir as the other containers — a mismatched workingDir in a newly added distro image produces exactly this empty-workspace symptom.
4. Re-run build LOG: Make sure main thread is getting id=0 #47; do not treat this as a blocker for PR UCT/ZE/COPY: Allocate DMA-BUF-exportable memory for RDMA #11903.

Related: PR #12094 (CI: Jenkins integration testing), PR #12166 (Jenkins CI: Gate Jenkins jobs on a basic compile), commit 9af60a8 / PR #11906 (new distro CI rows)

🛡️ This comment had 1 potential secret(s) redacted (Base64 High Entropy String). See request_id 1d9d59b1-1448-4a4e-b03a-5618e23f697f in the triage console for the audit trail.

[svc-ucx]

🤖 CI Triage Agent — UCX-OSS/ucx-test-gpu-oss · commit 346c1231

TL;DR: Three of the four GPU test shards hung in UCP RMA GET on host memory in the non-blocking memory-map variants (nb_no_rcache / map_nb), i.e. ODP (IBV_ACCESS_ON_DEMAND) registrations — the RDMA read never completed and gtest aborted after 3×180 s; this is not caused by PR #11903, whose code (ZE copy + a trace-only IB tweak) is never loaded on these CUDA nodes.

Full analysis

Summary: Test stages on funk06, funk13 and funk18 aborted with request ... did not complete on time → data validation failed (test_ucp_memheap.cc:115) → Connection timed out - abort testing (test_helpers.cc:57, signal 6), making the build fail with exit code 2.

Root cause: A silent, permanent stall of large RDMA GET operations against non-blocking (ODP) memory registrations. The three independent hangs share one exact ingredient:

• funk06 (stage 224): shm_ib_ipc/test_ucp_rma_reg.get_blocking/4 <shm,ib,cuda_ipc,rocm_ipc,cuda_copy,rocm_copy/**nb_no_rcache**> — last progress at size 415945, then silence 17:23:59 → 17:27:06.
• funk13 (stage 287): dcx/test_ucp_rma_reg.get_nonblocking/4 <dc_x,cuda_copy,rocm_copy/**nb_no_rcache**> — last progress at 306195, silence 17:44:18 → 17:47:24.
• funk18 (stage 302): rcx/test_ucp_rma.get_nonblocking/0 <rc_x,cuda_copy,rocm_copy/flush_worker> — hung inside the **map_nb** sub-case after 22041.

All three are the largest size class, all are GET, all use UCP_MEM_MAP_NONBLOCK. That flag maps to UCT_MD_MEM_FLAG_NONBLOCK → UCT_IB_MEM_FLAG_ODP → IBV_ACCESS_ON_DEMAND (src/uct/ib/base/ib_md.c:641-645, :671-673). ODP is advertised purely on the coarse device capability (uct_ib_md_check_odp() sets md->reg_nonblock_mem_types from IBV_DEVICE_HAS_ODP only, ib_md.c:1353-1380); no per-transport RC/DC rc_odp_caps RDMA_READ/WRITE bit is checked. If kernel page-fault resolution for the read doesn't complete, the QP yields no completion and no error — exactly the observed symptom (no WC error, no endpoint failure, no IB async event, just 3-minute silences). The large gaps (3 min, then 12 min to the abort) confirm a hang, not slowness, so this is not a timeout-tuning issue.

The PR is a red herring: its commits (346c1231, 4a2d4f44, 75f974d2, b402de77, and earlier 53c855b8/ac618d10) touch only src/uct/ze/* and, in src/uct/ib/base/ib_md.c, the shared DMA-BUF export probe (uct_ib_md_check_dmabuf, ib_md.c:1326-1351) plus the max_zcopy_log_sge condition (ib_md.c:1473-1481), which is only the packet-tracer SGE dump limit (ib_device.h:239). No ze_* transport appears in any test variant list (only cuda_copy/cuda_ipc/rocm_*), so the changed code is not even loaded on these nodes.

Implicated commit: unknown — not attributable to this branch; the hanging path is master's ODP/non-blocking registration advertisement (last touched by b6b85316, tomerg-nvidia, "UCP/UCT/UCS: Add memory registration flags", which introduced the reg_nonblock_mem_types plumbing)

File: src/uct/ib/base/ib_md.c:1368-1380 (uct_ib_md_check_odp) and src/uct/ib/base/ib_md.c:641-645 (NONBLOCK → ODP flag); failure observed at test/gtest/ucp/test_ucp_memheap.cc:115

Suggested fix:

1. Re-run the three failed shards to confirm; this is almost certainly not a PR regression, so don't block UCT/ZE/COPY: Allocate DMA-BUF-exportable memory for RDMA #11903 on it.
2. Confirm the ODP diagnosis by re-running the GPU job with non-blocking registration disabled, e.g. UCX_IB_ODP_MEM_TYPES= (empty) or UCX_IB_ODP_PREFETCH=y. If test_ucp_rma_reg.*_nb* stops hanging, ODP is the culprit.
3. Check fabric/driver health on funk06/funk13/funk18: ibv_devinfo -v | grep -i odp, MLNX_OFED/nvidia_peermem version, and dmesg for ODP page-fault errors. (Corroborating evidence of a degraded node: shm_ib/test_ucp_tag_match_rndv_align.recv_align/1 took 35.6 s on funk13.)
4. Hardening in UCX: in uct_ib_md_check_odp(), require the per-transport ODP capability bits (odp_caps.per_transport_caps.rc_odp_caps / dc_odp_caps with IBV_ODP_SUPPORT_READ|WRITE) before setting md->reg_nonblock_mem_types, so UCP transparently falls back to blocking registration instead of posting a read that can never complete.

Related: PR #11903 (the branch under test, unrelated to the failure); #12176 "GTEST/UCT: Skip tests which assume GDR capable memory under MPS"; no existing issue found for this ODP/non-blocking RMA GET hang — worth filing one with these three stage logs attached.

[svc-ucx]

🤖 CI Triage Agent — UCX-OSS/ucx-test-dl-gpu-oss · commit 346c1231

TL;DR: All four GPU test shards of this job die on the same underlying problem — ibv_reg_mr() returns EFAULT ("Bad address") for every cuda-managed buffer on the gb-nvl-* GB200 nodes, which cascades into 64 test failures, data-validation errors, hangs and Connection timed out - abort testing; this is a GPUDirect-RDMA/environment problem, not a regression from the ZE-only PR #11903.

Full analysis

Summary: UCX-OSS/ucx-test-dl-gpu-oss #61: all 4 parallel Test stages (nodes gb-nvl-059-compute07, gb-nvl-057-compute05/06, gb-nvl-059-compute06) failed — IB memory registration of CUDA managed memory fails with Bad address, followed by hangs and aborts.

Root cause: The common, reproducible error signature across shards is:

ib_md.c:324 UCX ERROR ibv_reg_mr(address=0xfffce0021000, length=8, access=0xf) failed: Bad address
/ucx/.../test/gtest/uct/uct_test.cc:1063: Failure  Error: Input/output error

emitted every time a test switches from memory_type:host to memory_type:cuda-managed (e.g. rc_verbs/uct_p2p_am_test.am_bcopy/1, am_zcopy/1, am_short_iov_keep_data/1). EFAULT from ibv_reg_mr on a cudaMallocManaged pointer means the kernel peer-memory path (nvidia_peermem) / dmabuf export is not available for managed memory on these nodes, yet the IB MD still advertises reg_mem_types for it. Everything else in the run is downstream of that:

• Shard 412 completed with 64 FAILED: ib/test_md.reg/3, reg/7, all uct_p2p_am_test.am_*_keep_data, uct_p2p_rma_test.{put_bcopy,get_zcopy}, test_ucp_mmap.rndv_mpool_mdesc_no_rcache/2, test_ucp_rma*_gpu_nic, and the whole rc_gda/test_ucp_device* family — all GPU-memory registration dependent.
• Shard 358: same ibv_reg_mr ... Bad address, then ib/test_ucp_rma_rndv.get_blocking/0 → ucp_test.cc:310 request ... did not complete on time, test_ucp_memheap.cc:115 data validation failed, then abort.
• Shard 361: fatal ucp_request.inl:356 Fatal: unexpected error: Request canceled inside uct_rc_txqp_purge_outstanding right after the cuda RMA test (ib/test_ucp_rma_rndv_cuda_async.put_nonblocking/1).
• Shard 381: a genuine hang — last output [ RUN ] dcx/test_ucp_tag_mt.send_recv/0 <dc_x/req_int,mt_context> at 17:34:28, then 15 minutes of silence to 17:49:35 when test_helpers.cc:57 Connection timed out - abort testing fired; the backtrace is in libgomp (OpenMP worker blocked in syscall), i.e. the MT DC tag send/recv never progressed. The abort/exit-code-2 is the symptom, the MT DC hang is the thing to investigate separately.

The triggering PR (#11903, fix/ze-dmabuf-exportable-allocation) only touches src/uct/ze/* (Level-Zero, not compiled in this CUDA/aarch64 image), plus a new shared test helper mem_buffer::query_dmabuf_export() and test/gtest/uct/test_md.cc. None of that is on the CUDA/IB registration path, so it is not causally implicated.

Implicated commit: unknown (not PR #11903; its commits — 346c123 / 4a2d4f4, Yaser Afshar — only touch ZE code and test helpers). Node/driver configuration on the gb-nvl GB200 partition is the trigger.

File: src/uct/ib/base/ib_md.c:324 (error report site); guard helper already available at test/gtest/common/mem_buffer.cc:381 (mem_buffer::cuda_mem_is_gdr_capable)

Suggested fix:

1. Infra: on the gb-nvl-0xx compute nodes verify nvidia_peermem is loaded and that dmabuf export works for managed allocations (cuPointerGetAttribute(CU_POINTER_ATTRIBUTE_IS_GPU_DIRECT_RDMA_CAPABLE) is evidently 0 there, likely because the job runs under MPS). Without that, no IB registration of CUDA/managed memory can succeed and this job will keep failing for every PR.
2. Code-side hardening (already in flight upstream): don't advertise IB registrability for non-GDR-capable memory — PR UCT/CUDA_COPY: Non GDR capable memory is non-registrable without DMABUF #12055 — and skip the managed-memory UCT reg tests when mem_buffer::cuda_mem_is_gdr_capable() is false, as done by GTEST/UCT: Skip tests which assume GDR capable memory under MPS MLOPart #12176. Merging those will turn these 64 hard failures into skips.
3. Do not raise the stage timeout for shard 381: the 15-minute gap after dcx/test_ucp_tag_mt.send_recv/0 is a hang in the multi-threaded DC tag path, which deserves its own bug (reproduce with --gtest_filter=dcx/test_ucp_tag_mt.send_recv/0 plus UCX_LOG_LEVEL=debug).
4. Re-run UCP/GTEST: initialization of the ucp layer. and a test in gtest for testing it. #61 once the nodes are fixed to confirm PR UCT/ZE/COPY: Allocate DMA-BUF-exportable memory for RDMA #11903 is clean; nothing in its diff can produce these failures.

Related: #11903 (triggering PR), #12055 (non-GDR memory non-registrable without dmabuf), #12176 (skip tests assuming GDR-capable memory under MPS), #12263 (managed RNDV frag pool blocking registration), #11885 (moved this job to the GB200 partition)

[yafshar]

@tvegas1 Thanks for kicking off /build. ucx-codestyle-oss flags clang-format on the changed lines. I can push a format-only fix for the ternary and test hunks; the config table hunk would stay, since clang-format reflows the whole ucs_config_field_t initializer and I've kept it in the same layout as the other config tables. Would you like me to push that now (it restarts CI and needs a re-approval), or merge as is and I follow up with a format-only PR?

The other failures look unrelated to this change: the rcache assert in tcp/test_ucp_rma.put_nonblocking (host memory over TCP), the BlueField 6 h timeout, and the *-oss build/GPU jobs, per the triage bot. If you prefer merging as is, the "Tests new on worker 1" job needs a rerun once build 143160 finishes.