[mshanthagit]

What?

Advertises peer failure error handling capability

Why?

How?

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

rocm_ipc never calls uct_iface_handle_ep_err() (see rocm_ipc_ep.c:95-160, rocm_signal.c:57-63) and returns UCS_ERR_INVALID_ADDR/UCS_ERR_IO_ERROR, which are not endpoint-level errors (UCS_IS_ENDPOINT_ERROR). Even without the fatal above, UCP would fail the individual request rather than mark the lane/EP failed, so the flag does not deliver peer-failure semantics. How is a dead peer expected to be detected and reported to UCP?

[mshanthagit]

wouldn't UCP's keepalive lane detect the dead peer?

[svc-nvidia-pr-review]

rocm_ipc MD (rocm_ipc_md.c:33) reports only REG | NEED_RKEY | IPC_MEMTYPE_COPY, while cuda_ipc additionally reports INVALIDATE | INVALIDATE_RMA | INVALIDATE_AMO. With err_mode=peer, ucp_wireup_add_rma_bw_lanes() adds UCT_MD_FLAG_INVALIDATE_RMA to the local MD criteria (ucp/wireup/select.c:2278-2280), and rocm_ipc is CONNECT_TO_IFACE so the p2p exemption at select.c:570-574 does not apply. Will rocm_ipc actually be selected as an rndv/RMA lane with error handling, or is the iface flag alone not enough (i.e. the change may be a no-op for the intended use case)?

[mshanthagit]

added the required flags

[svc-nvidia-pr-review]

Test/validation gap: nothing exercises rocm_ipc with a dead peer — test_uct_peer_failure requires AM_SHORT, test_uct_purge_outstanding requires V2_QUERY_TOKEN, test_uct_peer_failure_rma_zcopy is instantiated for cma only, and the uct_flush_test CANCEL branches all need AM/BCOPY caps. How was the new capability verified on ROCm hardware?

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

How does a dead peer reach the user's err_handler? Nothing in rocm/ipc calls uct_iface_handle_ep_err(), and the attach failure now returns UCS_ERR_INVALID_ADDR, which is not an endpoint error, so UCP aborts the request via ucp_proto_request_abort() and leaves the ep alive. With this flag set, wireup will pick rocm_ipc for RMA lanes in UCP_ERR_HANDLING_MODE_PEER.

[svc-nvidia-pr-review]

Test/validation gap: no test exercises the newly advertised capability. Every candidate test is skipped for rocm_ipc: test_uct_peer_failure requires AM_SHORT|CB_SYNC, test_uct_purge_outstanding requires V2_QUERY_TOKEN, test_uct_peer_failure_keepalive requires EP_CHECK (which this PR stubs to UNSUPPORTED), and test_uct_peer_failure_rma_zcopy is instantiated via _UCT_INSTANTIATE_TEST_CASE(..., cma) only.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 3) · commit 019c6ca9

TL;DR: The gtest run itself was 100% green (8707/8707 passed) — the job failed only because LeakSanitizer reported a single 40-byte leak allocated inside libmlx5.so via the DEVX CQ umem registration path, which is unrelated to this PR's ROCm IPC change; re-run the job, and harden uct_ib_mlx5_md_buf_free() to check the mlx5dv_devx_umem_dereg() return code (or add an LSan suppression for rdma-core internal allocations).

Full analysis

Summary: AddressSanitizer roce on worker 3 failed at the very end of the gtest stage with ==26929==ERROR: LeakSanitizer: detected memory leaks → SUMMARY: AddressSanitizer: 40 byte(s) leaked in 1 allocation(s), causing make: *** [Makefile:4809: test] Error 1.

Root cause: All tests passed ([ PASSED ] 8707 tests.), so this is a process-exit leak check failure, not a test assertion. The leaking allocation is an internal rdma-core object:

Direct leak of 40 byte(s) in 1 object(s) allocated from:
  #0 __interceptor_calloc
  #1 0x7feaa78ce9de (/lib/x86_64-linux-gnu/libmlx5.so.1+0x6b9de)
  #2 uct_ib_mlx5_md_buf_alloc   src/uct/ib/mlx5/ib_mlx5.h:1186
  #3 uct_ib_mlx5_devx_create_cq src/uct/ib/mlx5/dv/ib_mlx5_dv.c:683
  #4 uct_ib_iface_t_init        src/uct/ib/base/ib_iface.c:1888
  #5 uct_rc_iface_t_init ... #9 uct_iface_open
  #10 ucp_worker_iface_open     src/ucp/core/ucp_worker.c:1572
  #13 ucp_test_base::entity::entity ... test_ucp_rma_sgl::init()

Frame #1 is inside libmlx5, i.e. the 40 bytes are the mlx5dv_devx_umem_reg() bookkeeping struct allocated by rdma-core, not UCX memory. UCX's own page-aligned CQ buffer (cq->devx.cq_buf) is not reported as leaked, which means uct_ib_mlx5_md_buf_free() did run and ucs_free(buf) succeeded — but the paired mlx5dv_devx_umem_dereg(mem->mem) evidently did not release the rdma-core struct (its return value is discarded, src/uct/ib/mlx5/ib_mlx5.h:1184), so the failure is silently swallowed and shows up only as an LSan report at exit. Nothing in the PR under test (#11977, "UCT/ROCM_IPC: Advertise peer failure error handling capability") touches the IB/mlx5 DEVX CQ path, so this is a pre-existing/environment-dependent leak in the mlx5 DEVX CQ teardown, not a regression from the commit being validated.

Implicated commit: unknown — no commit from this PR appears in the leak stack; the implicated code (uct_ib_mlx5_md_buf_alloc/_free, DEVX CQ create/destroy) was last touched by unrelated changes (e.g. 5f3c75da Artemy Kovalyov "UCT/MLX5: Add DEVX QP/CQ on foreign memory support", [REDACTED:Hex High Entropy String] nbellalou "UCT/IB/MLX5: Fix DEVX for not WC-supported UAR").

File: src/uct/ib/mlx5/ib_mlx5.h:1184 (mlx5dv_devx_umem_dereg return value ignored in uct_ib_mlx5_md_buf_free); allocation site src/uct/ib/mlx5/dv/ib_mlx5_dv.c:676-683, teardown src/uct/ib/mlx5/dv/ib_mlx5_dv.c:713-718.

Suggested fix:

1. Short term: re-run the AddressSanitizer roce job — this failure is not attributable to PR UCT/ROCM_IPC: Advertise peer failure error handling capability #11977 and should not block it.
2. Make the failure diagnosable instead of silent — check the dereg return code in uct_ib_mlx5_md_buf_free():

   ret = mlx5dv_devx_umem_dereg(mem->mem);
   if (ret != 0) {
       ucs_warn("mlx5dv_devx_umem_dereg(umem=%p size=%zu) failed: %m",
                mem->mem, mem->size);
   }

   A warning here will immediately confirm whether the 40 bytes are leaked because dereg failed (likely a teardown-ordering issue where the DEVX CQ object still references the umem) rather than leaving only an LSan report at exit.
3. Verify teardown ordering in uct_ib_mlx5_devx_destroy_cq() (ib_mlx5_dv.c:713): the DEVX CQ object must be fully destroyed before its umem is dereg'd; if uct_ib_mlx5_devx_obj_destroy(cq->devx.obj, "CQ") can fail/be skipped, the umem dereg will fail too and leak exactly this struct.
4. If this proves to be an unfixable rdma-core-internal artifact, add a targeted LSan suppression (leak:libmlx5) to the ASAN job config so a 40-byte third-party allocation cannot red-flag an otherwise fully green 8707-test run.

Related: PR #11977 (UCT/ROCM_IPC: Advertise peer failure error handling capability) — the PR under test; no existing issue found for this leak signature.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 7c95d014-8898-496a-b736-43e537e0cb44 in the triage console for the audit trail.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (AddressSanitizer gpu on worker 3) · commit 019c6ca9

TL;DR: The ASAN-GPU gtest run failed on a single test, shm_ib/test_ucp_proto_am_rndv.am_explicit_scheme_disables_force_provenance/0, which asserts a specific protocol-selection outcome (am/rndv + zero UCP_RKEY_CONFIG_FLAG_PROTO_ESTIMATION rkey configs) that is topology/host dependent — it is unrelated to PR #11977's ROCm-IPC capability change (rocm_ipc isn't even in the failing variant's TL list). Re-run the job and track the test against the known provenance/topology issue (#11990); do not treat this as a regression from this PR.

Full analysis

Summary: make test in build-test/test/gtest exited with Error 1 because exactly 1 of 10336 tests failed: shm_ib/test_ucp_proto_am_rndv.am_explicit_scheme_disables_force_provenance/0, where GetParam() = shm,ib,cuda_copy,rocm_copy.

Root cause: Evidence from the log: [ PASSED ] 10335 tests and [ FAILED ] 1 test ... am_explicit_scheme_disables_force_provenance/0, no sanitizer report, no crash, no timeout (test suite completed normally in 2073984 ms and tore down cleanly). The failing test (test/gtest/ucp/test_ucp_proto.cc:805-813) runs with RNDV_PIPELINE_SHM_CUDA_STAGING_FORCE=y + RNDV_SCHEME=get_zcopy and calls check_am_rndv_remote_proto_config(0, 0), which hard-asserts three environment-sensitive things: that am/rndv is the protocol selected at 1 MB for CUDA memory (EXPECT_STREQ("am/rndv", ...), line 438), that the remote op_id_flags equal exactly UCP_OP_ID_RNDV_RECV (line 455), and that zero rkey configs carry UCP_RKEY_CONFIG_FLAG_PROTO_ESTIMATION (line 457). Those outcomes depend on lane/MD reachability and system-device distance computed in ucp_proto_rndv_ctrl_get_md_map() (src/ucp/rndv/proto_rndv.c:112, ucs_topo_is_reachable) and on the remote sys_dev proxy logic added at src/ucp/rndv/proto_rndv.c:370-381, so the assertion can flip on a specific GPU worker's topology. The sibling tests with identical expectations but without RNDV_SCHEME=get_zcopy (am_default_remote_model_preserves_legacy_key, am_force_remote_model_remains_untagged) passed in the same run, consistent with a selection/topology-sensitive assertion rather than a code fault. PR #11977 ("UCT/ROCM_IPC: Advertise peer failure error handling capability") cannot plausibly affect this variant: the failing parameterization is shm,ib,cuda_copy,rocm_copy — rocm_ipc is not loaded.

Implicated commit: Not this PR. The assertion and its remote-provenance model come from the CUDA staging-force series — b34ac9a1 / 9cc08081 / 8a76f891 (Nathan Bellalou) — and the remote sys_dev isolation in 0311ed79 (jeffnvidia, "UCP/PROTO/RNDV: Isolate remote system device namespace", #11685). Tested commit [REDACTED:Hex High Entropy String] is not implicated.

File: test/gtest/ucp/test_ucp_proto.cc:805 (test body) and test/gtest/ucp/test_ucp_proto.cc:438-458 (check_am_rndv_remote_proto_config assertions); selection logic at src/ucp/rndv/proto_rndv.c:370-381 and src/ucp/rndv/proto_rndv.c:112.

Suggested fix: Re-run the "AddressSanitizer gpu on worker 3" job to confirm the failure is not reproducible with this PR; it should not block #11977. For a durable fix, make the test tolerant of hosts where am/rndv is not the winning protocol for CUDA at 1 MB under RNDV_SCHEME=get_zcopy: in select_am_rndv_remote_proto_config() replace the hard EXPECT_STREQ("am/rndv", ...) with a UCS_TEST_SKIP_R("am/rndv not selected") when another protocol wins (mirroring the existing "stub protocol" skip at line 700-702), and/or add a topology guard alongside require_cuda_rndv_support() that skips when no intra-node CUDA-capable lane with the required reachability exists. If the worker in question has a DPU/BlueField in its topology, pick up the fix from #11990 ("UCP/CORE: Ignore DPU in topology assignments").

Related: PR #11977 (the PR under test), #11990 (UCP/CORE: Ignore DPU in topology assignments), #11685 (remote system device namespace isolation), and the diagnostic PRs opened specifically against this test: #11858, #11861, #11864.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id fa28ae98-bf11-4e4f-80b2-bd32027d0a8e in the triage console for the audit trail.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

uct_rocm_ipc_mem_dereg() still passes 0 as _invalidate_supported and ignores UCT_MD_MEM_DEREG_FLAG_INVALIDATE. With these flags advertised, rocm_ipc gets into memh->inv_md_map and ucp_memh_dereg() passes that flag, so with params check enabled dereg returns UCS_ERR_UNSUPPORTED before ucs_free(mem_hndl) (key leak), and otherwise the invalidation is skipped and comp is never invoked. How is invalidation supposed to work for rocm_ipc?

[svc-nvidia-pr-review]

Residual coverage gap: no new test exercises the newly advertised capability, and nothing in the diff covers the mem_dereg(INVALIDATE) path for rocm_ipc, which is why the flag/implementation mismatch isn't caught by CI.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 0) · commit 1cf9701f

TL;DR: The ASAN/RoCE gtest shard hung on its very first test, dcx/test_ucp_am_nbx_rndv.dts/2 <dc_x/prereg> — the 64KB rendezvous transfer over the dc_x lane never made progress (0 of 1 messages received) for ~15 min until the 900 s gtest watchdog SIGABRT'ed the process; this is a transport/fabric-level stall unrelated to PR #11977 (which only touches rocm_ipc, not used by this host-memory dc_x test).

Full analysis

Summary: make -C test/gtest test aborted (core dumped) because dcx/test_ucp_am_nbx_rndv.dts/2 <dc_x/prereg> never completed; request ... did not complete on time followed by the watchdog's "Connection timed out - abort testing".

Root cause: Hang, not slowness. The log shows a single 877-second gap with zero output between [ RUN ] dcx/test_ucp_am_nbx_rndv.dts/2 <dc_x/prereg> (17:09:20) and the first failure message (17:23:57) — the entire runtime of the test. In that window the AM rendezvous send was posted (m_send_counter = 1) but the receiver never got any data (m_recv_counter = 0), and ucp_test::request_process() (test/gtest/ucp/ucp_test.cc:300) expired at its ASAN-scaled deadline (watchdog_get_timeout() * 0.75 = 675 s, see test/gtest/common/test_helpers.cc:342-347). The gtest watchdog then fired at its 900 s default (test/gtest/common/test_helpers.cc:31,57-59) and sent SIGABRT, producing the Caught signal 6 backtrace whose stack is simply usleep inside the test's progress/wait loop — i.e. the test thread was still spinning waiting for a completion that never arrived. No UCX transport error, peer-failure, or ASAN report was emitted, so the dc_x lane on swx-rain03's RoCE device silently stopped delivering/completing the rendezvous GET — a fabric/DC-connectivity stall, not an application-level error path.

Crucially, the test under which this happened uses host memory over dc_x only. The commit under test (1cf9701f, "UCT/ROCM_IPC: Add peer failure error handling", Manu Shantharam) only adds UCT_IFACE_FLAG_ERRHANDLE_PEER_FAILURE to the ROCm IPC iface caps (src/uct/rocm/ipc/rocm_ipc_iface.c:124) — rocm_ipc is not instantiated for this dcx/... variant and cannot influence DC rendezvous completion. The failure is therefore infrastructural/flaky with respect to this PR.

Implicated commit: unknown (the PR commit 1cf9701f by Manu Shantharam is not implicated — it touches only src/uct/rocm/ipc/*, which is not exercised by the dc_x host-memory variant that hung)

File: test/gtest/ucp/test_ucp_am.cc:542 (assertion site, test_am_send_recv) / test/gtest/ucp/ucp_test.cc:300 (deadline expiry) — the hang itself is in the dc_x rendezvous path, not in these files

Suggested fix: Re-run the "AddressSanitizer roce on worker 0" job; this is not a code defect introduced by the PR. If it reproduces on the same agent, treat it as node health: check swx-rain03's RoCE device/DC state (link up, GID/RoCE v2 config, ibv_devinfo, DC key errors, stuck QPs, leftover gtest processes from a previous aborted shard holding the HCA). To make future occurrences diagnosable rather than a blind 15-minute silence, capture UCX_LOG_LEVEL=info plus a ucp_worker_print_info/VFS dump when request_process() hits its deadline in test/gtest/ucp/ucp_test.cc:298-301, so the stalled lane and protocol are logged before the watchdog aborts. Do not raise the watchdog timeout — the transfer made zero progress, so more time would not have helped.

Related: PR #11977 (the PR under test, ROCm IPC peer-failure capability — unrelated to the hang). No existing issue found matching test_ucp_am_nbx_rndv + dc_x rendezvous stall; searches for "test_ucp_am_nbx_rndv dts request did not complete on time" and "dc_x prereg rndv hang" returned no matches, so consider opening a flaky-test tracking issue if it recurs.

[edgargabriel]

I approve the PR as is. From the error handling capability, the commit matches the rocm_ipc TL with the cuda_ipc. Neither one truly handles every situations correctly.

The flag is however required for the rocm_ipc component to be used from NIXL, and the PR adds some base capabilities. Anything beyond the PR can be addressed at a later stage if necessary,