[jeynmann]

What?

Add wqe parse support for uct get.

Depends on #11874

Why?

FT need extract info from wqe before replay uct get.

How?

Add parser for uct get bcopy/zcopy.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

src/uct/ib/mlx5/rc/rc_mlx5_common.c

uct_rc_mlx5_op_info_fill assumes no AV segment between ctrl and raddr — only correct for RC, not DC. raddr = (void*)(ctrl + 1) and wqe_size - sizeof(*ctrl) - sizeof(*raddr) do not account for the datagram/AV segment that DC WQEs place after the ctrl segment. The helper lives in the RC/DC common file and is exported, but it takes no av_size. Since the new gtest only instantiates rc_mlx5, this is not currently exercised for DC, but the naming/placement implies shared use. Question for the author: is DC intended to be supported by this helper, and if so should it take av_size like the post path (UCT_RC_MLX5_RMA_MAX_IOV(UCT_IB_MLX5_AV_FULL_SIZE))?

Coverage note: The gtest (test_rc_purge_outstanding) is instantiated for rc_mlx5 only (_UCT_INSTANTIATE_TEST_CASE(..., rc_mlx5)), so dc_mlx5 and the AV-segment path are not covered. If DC support is intended, a DC instantiation would be needed once the AV-segment finding is resolved.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (Tests BlueField on worker 1) · commit 174dcfdf

TL;DR: The new gtest rc_mlx5/test_rc_purge_outstanding.get_zcopy fails because it issues a 64-byte uct_ep_get_zcopy(), which RC/mlx5 rejects with UCS_ERR_INVALID_PARAM (minimum GET-zcopy length is max_inl_cqe[TX]+1 = 65, since RC defaults IB_TX_INLINE_RESP=64); fix the test to use cap.get.min_zcopy (e.g. ≥65) instead of a hard-coded 64.

Full analysis

Summary: Azure job "BlueField on worker 1" failed in make test (gtest): [ FAILED ] rc_mlx5/test_rc_purge_outstanding.get_zcopy/0, where GetParam() = rc_mlx5/mlx5_0:1 → make: *** [Makefile:4713: test] Error 1 → Bash exited with code '2'. 8667/8668 tests passed; only this newly added test failed.

Root cause: The test added by this PR calls test_get(UCT_EP_OP_GET_ZCOPY, 64) and asserts ASSERT_EQ(UCS_INPROGRESS, uct_ep_get_zcopy(...)). For RC/mlx5, uct_rc_mlx5_base_ep_get_zcopy() enforces a minimum length:

• src/uct/ib/mlx5/rc/rc_mlx5_ep.c:385 — UCT_CHECK_LENGTH(total_length, iface->super.super.config.max_inl_cqe[UCT_IB_DIR_TX] + 1, iface->super.config.max_get_zcopy, "get_zcopy")
• src/uct/ib/rc/base/rc_iface.c:31 — RC overrides the IB defaults with TX_INLINE_RESP=64, so max_inl_cqe[UCT_IB_DIR_TX] is 64 and, per src/uct/ib/rc/base/rc_iface.c:234, cap.get.min_zcopy = 65 (reads ≤64B are scattered inline into the CQE and therefore cannot be zcopy).

So a 64-byte get_zcopy returns UCS_ERR_INVALID_PARAM (plus an unexpected ucs_error from UCT_CHECK_LENGTH, which gtest also flags), and the ASSERT_EQ aborts the test. The sibling get_bcopy case (8 bytes) has no such minimum, which is why only the zcopy variant failed. The WQE-parsing code itself (uct_rc_mlx5_op_info_fill_get_zcopy) is never reached. Note test_gga_get_zcopy_purge can use 64 bytes because gga_mlx5 uses the MMO path with no inline-CQE minimum — likely where the constant was copied from.

Implicated commit: 8e578f4d "GTEST/UCT/IB: test get" (Zihao Zhao), later refactored in eed21d92; branch failover_uct_rcx_get, HEAD 174dcfdf.

File: test/gtest/uct/ib/test_rc.cc:1735 (and the assert it triggers at test/gtest/uct/ib/test_rc.cc:1707)

Suggested fix: Derive the zcopy length from the interface attributes instead of hard-coding it, e.g.:

UCS_TEST_SKIP_COND_P(test_rc_purge_outstanding, get_zcopy,
                     !check_caps(UCT_IFACE_FLAG_GET_ZCOPY))
{
    test_get(UCT_EP_OP_GET_ZCOPY,
             ucs_max(64ul, m_e1->iface_attr().cap.get.min_zcopy));
}

(any length > max_inl_cqe[UCT_IB_DIR_TX], e.g. 4096, works; alternatively add "IB_TX_INLINE_RESP=0" to the test's config). Optionally add an EXPECT_GE(length, iface_attr().cap.get.min_zcopy) guard in test_get() so the constraint is explicit.

Related: PR #11866 (this change); sibling PRs #11836 (put parser) and #11844 (flush parser) use the same helper and may need the same length guard.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

Residual coverage note: the gtest instantiates rc_mlx5 only; dc_mlx5 and the AV-segment path are untested. Also, only MLX5_OPCODE_RDMA_READ (GET) is implemented — uct_rc_mlx5_op_info_fill ucs_fatal's on any other opcode, so PUT/AM/ATOMIC purge paths are not yet handled; confirm that matches the intended scope of this PR.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (Build build_aarch64 rocky89_aarch64) · commit 7a7881cf

TL;DR: The aarch64 build failed to compile test/gtest/uct/ib/test_rc.cc because line 1730 references iface_attr().cap.get.min_bcopy, a field that does not exist in uct_iface_attr (only min_zcopy exists for GET); drop the min_bcopy term from the new get_bcopy test.

Full analysis

Summary: make failed in test/gtest — uct/ib/gtest-test_rc.o compile error (-Werror/hard error), aborting the whole build.

Root cause: New gtest added by this PR uses a non-existent struct member:

test_rc.cc:1730:62: error: 'const struct uct_iface_attr::<unnamed struct>::<unnamed>'
  has no member named 'min_bcopy'; did you mean 'min_zcopy'?
      ucs_min(ucs_max(8ul, m_e1->iface_attr().cap.get.min_bcopy),

struct uct_iface_attr::cap::get (src/uct/api/uct.h:1084-1099) defines only max_short, max_bcopy, min_zcopy, max_zcopy, opt_zcopy_align, align_mtu, max_iov. There is no min_bcopy for bcopy operations (bcopy has no minimum length), so the test_rc_purge_outstanding.get_bcopy test body doesn't compile. This is not aarch64-specific — it will fail on every builder.

Implicated commit: [REDACTED:Hex High Entropy String] "GTEST/UCT/IB: test get" (Zihao Zhao), further touched by bf18d404 "GTEST/UCT/IB: refactor" (Zihao Zhao), on branch failover_uct_rcx_get (PR #11866)

File: test/gtest/uct/ib/test_rc.cc:1730

Suggested fix: Remove the bogus minimum-bcopy lookup and just clamp the fixed length to the max:

UCS_TEST_SKIP_COND_P(test_rc_purge_outstanding, get_bcopy,
                     !check_caps(UCT_IFACE_FLAG_GET_BCOPY))
{
    test_get(UCT_EP_OP_GET_BCOPY,
             ucs_min(8ul, m_e1->iface_attr().cap.get.max_bcopy));
}

(The get_zcopy variant at line 1738 is correct as-is since cap.get.min_zcopy does exist.) If a minimum really is intended for bcopy, the field would first have to be added to uct_iface_attr and populated by all transports — not recommended.

Related: PR #11866 (this PR). No existing issue found referencing cap.get.min_bcopy.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 0d7baa68-d978-4922-ac49-5233ade0a331 in the triage console for the audit trail.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

Residual coverage note (matches prior reviews): the gtest instantiates rc_mlx5 only, so dc_mlx5 / the AV-segment path is untested, and only MLX5_OPCODE_RDMA_READ (get) is exercised — uct_rc_mlx5_op_info_fill ucs_fatals on any other opcode.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[gleon99]

As discussed, please replace with a stress test.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (Tests roce on worker 2) · commit da77720d

TL;DR: The roce on worker 2 gtest run failed on a single test, ud/test_ucp_peer_failure.zcopy/3 <ud_v/rma_fail_imm> (8714/8715 passed); this is a long-standing flaky peer-failure test on the UD transport and is unrelated to the mlx5 GET WQE-parser changes in PR #11866 — retrigger the job and track the flake in the existing issues.

Full analysis

Summary: make test in build-test/test/gtest exited with code 1 because exactly one gtest case failed: ud/test_ucp_peer_failure.zcopy/3, where GetParam() = ud_v/rma_fail_imm.

Root cause: From the log, the run completed normally (8715 tests, 3495 s, continuous output, no hang, no timeout, no crash/signal, no sanitizer report) with a single failing case: test_ucp_peer_failure.zcopy on the ud_v transport with the rma_fail_imm variant. This test deliberately kills the receiver entity and then asserts on error-status timing (test/gtest/ucp/test_ucp_peer_failure.cc:392 EXPECT_EQ(UCS_OK, m_err_status), :409 EXPECT_NE(UCS_OK, m_err_status), :418-429 status/err-status equality checks). Because UD implements reliability in software with a 3 s peer timeout artificially forced by the test (set_ib_ud_peer_timeout(3.), line 397), the exact moment the error callback fires relative to request completion is load-dependent, which makes these assertions race-prone on a busy CI worker. The failing configuration exercises only ud_verbs (UCP_INSTANTIATE_TEST_CASE → ud TLS group) and the RMA/put path — not the rc_mlx5/dc_mlx5 GET zcopy WQE path that PR #11866 ("[DNM] UCT/IB/MLX5: wqe parser for uct get") modifies, so the change under test is not on the code path that failed. The same test/variant has failed historically in this exact form (see Related), confirming it as a pre-existing flake rather than a regression.

Implicated commit: unknown — no evidence links [REDACTED:Hex High Entropy String] (branch failover_uct_rcx_get, PR #11866) to the failing ud_v code path; the most recent functional change to the test file is dad9eb55 (nbellalou, "UCP/CORE: Verify rcache validity before calling ucp_memh_invalidate", #11485), which predates this PR.

File: test/gtest/ucp/test_ucp_peer_failure.cc:392-429 (assertions in do_test), test instantiation at test/gtest/ucp/test_ucp_peer_failure.cc:597 (zcopy)

Suggested fix:

1. Retrigger the roce on worker 2 job to confirm the failure is not reproducible; a single-test failure on an untouched transport should not block PR UCT/IB/MLX5: support rma get in purge outstanding #11866.
2. To make the diagnosis conclusive, capture the actual gtest assertion text — the Azure step log is tail-truncated past the failure point. Re-run with GTEST_FILTER='ud/test_ucp_peer_failure.zcopy/3' and UCX_LOG_LEVEL=debug (e.g. via contrib/test_jenkins.sh with a reduced filter) so the [ FAILED ] block with file:line is visible.
3. To de-flake the test itself: replace the point-in-time checks around lines 409-429 with a bounded wait for the error status to stabilize (poll m_err_status/request_wait until ucs::get_deadline()), rather than asserting on a single progress iteration, and consider raising the forced UD peer timeout (line 397) above 3 s for the FAIL_IMM variants where wireup has not yet completed.

Related: #6762 (udx/test_ucp_peer_failure.zcopy/3 failure, still open), #6203 (dcx/test_ucp_peer_failure.zcopy/3 failed, still open), #11866 (PR under test)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 4f0aa260-33dc-4844-b18b-568fecc67a79 in the triage console for the audit trail.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

DC/AV segment: uct_rc_mlx5_op_info_fill_get computes raddr = ctrl + 1 and wqe_size - sizeof(ctrl) - sizeof(raddr), which ignores the datagram/AV segment present on DC WQEs. The helper lives in the RC/DC common file and is exported but takes no av_size, and UCT_RC_MLX5_RMA_MAX_IOV(0) is used for the callback buffer. This matches an unresolved thread from earlier reviews.

Coverage: only MLX5_OPCODE_RDMA_READ (get) is implemented; uct_rc_mlx5_op_info_fill ucs_fatals on any other opcode, and the gtest is instantiated for rc_mlx5 only (no dc_mlx5, no AV-segment path). Confirm this matches intended PR scope.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

src/uct/ib/mlx5/rc/rc_mlx5_common.c

uct_rc_mlx5_op_info_fill_get computes raddr = ctrl + 1 and wqe_size - sizeof(ctrl) - sizeof(raddr), ignoring the datagram/AV segment that DC WQEs place after ctrl. The callback buffer is sized with UCT_RC_MLX5_RMA_MAX_IOV(0) while DC posts use UCT_RC_MLX5_RMA_MAX_IOV(UCT_IB_MLX5_AV_FULL_SIZE) (dc_mlx5_ep.c:596/658). So for DC the iov count can exceed callback_data->iov, and the ucs_assert(iovcnt <= ucs_static_array_size(...)) in uct_rc_mlx5_callback_data_fill_iov would be the only guard.

src/uct/ib/mlx5/rc/rc_mlx5_common.c:250

minor: the struct wraps iov/memh in a union { struct { ... }; } with a single member, which adds no value. Suggest dropping the union:

typedef struct {
uct_iov_t iov[UCT_RC_MLX5_RMA_MAX_IOV(0)];
uct_ib_mem_t memh[UCT_RC_MLX5_RMA_MAX_IOV(0)];
} uct_rc_mlx5_op_callback_data_t;

Coverage gap (also noted in prior reviews): no dc_mlx5 instantiation and no non-RDMA_READ opcode coverage; confirm this matches the intended PR scope.

Could not verify uct_ep_op_info_t::rma.rkey buffer is >= 8 bytes for the uct_ib_md_pack_rkey write; worth a spot check by the author if not already covered elsewhere.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (Build build_aarch64 rhel100_aarch64) · commit a6f916eb

TL;DR: The aarch64 build failed compiling test/gtest/uct/test_peer_failure.cc — purge_outstanding() is declared with one parameter but its body references with_rx_token and its caller passes two arguments; this is a semantic merge conflict between PR #11866 and master commit 2dbbd8c ("Purge outstanding work without an RX token", #12006). Fix by rebasing the PR on master and changing the signature to void purge_outstanding(purge_ctx *ctx, bool with_rx_token).

Full analysis

Summary: Build on aarch64 stage of build_aarch64 rhel100_aarch64 failed with -Werror C++ compile errors in the gtest UCT peer-failure test, aborting make (make[3]: *** [Makefile:2543: uct/gtest-test_peer_failure.o] Error 1).

Root cause: Two compile errors in the merged tree:

• test_peer_failure.cc:640: error: 'with_rx_token' was not declared in this scope; did you mean 'rx_token'?
• test_peer_failure.cc:703: error: no matching function for call to 'purge_outstanding(purge_ctx*, bool&)' — "candidate expects 1 argument, 2 provided", candidate at line 629.

Reading the source confirms the inconsistency: void purge_outstanding(purge_ctx *ctx) is declared at line 629 (old signature from the PR branch's base), but its body at line 640 branches on with_rx_token, and test_purge_outstanding() at line 703 calls purge_outstanding(&ctx, with_rx_token). The with_rx_token parameter/plumbing was introduced on master by 2dbbd8c4 (Evgeny Leksikov, 2026-10-04, "UCT/API: Purge outstanding work without an RX token (#12006)"), which landed after the PR's HEAD commit a6f916eb (2026-09-29). Git merged the two hunks textually without conflict, but the result does not compile — a classic semantic merge conflict. This is not a flake, not a timeout, and not an environment issue; it would fail on any architecture.

Implicated commit: [REDACTED:Hex High Entropy String] (Zihao Zhao, "UCT/IB/MXL5: add get support for outstanding purge") merged against 2dbbd8c4 (Evgeny Leksikov, "UCT/API: Purge outstanding work without an RX token", #12006)

File: test/gtest/uct/test_peer_failure.cc:629 (declaration), with errors reported at :640 and :703

Suggested fix: Rebase branch failover_uct_rcx_get onto current master so the PR picks up the new purge_outstanding contract, then update the helper signature to match the post-#12006 API:

-    void purge_outstanding(purge_ctx *ctx)
+    void purge_outstanding(purge_ctx *ctx, bool with_rx_token)

Verify any other purge_outstanding() / test_purge_outstanding() call sites added by this PR (e.g. the new GET-support test cases) pass the with_rx_token argument explicitly, since test_purge_outstanding() still defaults it to true at line 674 and a PR-added caller may silently rely on the old one-arg form. Re-run the aarch64 debug build locally (-Werror is enabled) before pushing.

Related: PR #11866 (this PR); #12006 (2dbbd8c4, the API change that introduced with_rx_token); prior purge-outstanding series #11874, #12019, #11915, #11836

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id e83ed0e8-5a56-49f6-82c1-6296ea5ff09a in the triage console for the audit trail.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 3) · commit 36de2d26

TL;DR: The gtest shard hung for ~14.6 minutes inside shm_ib/test_ucp_am_nbx_closed_ep.rx_short_am_on_closed_ep/2 <shm,ib/reply> until the 900 s gtest watchdog fired SIGTERM/abort; the stall is in the RC/mlx5 error-handling + outstanding-purge path that this PR's HEAD commit 36de2d26 ("add get support for outstanding purge") rewrites, where read credits / send-ops for RDMA_READ are released on a path that never completes, so the AM send request never finishes.

Full analysis

Summary: make -C test/gtest test (ASan, roce, shard 4 of 4) aborted: request 0x62e00000a418 did not complete on time in test_ucp_am_nbx_closed_ep.rx_short_am_on_closed_ep/2, followed by the watchdog's "Connection timed out - abort testing" and signal 6.

Root cause: This is a hang, not a slow test. Timestamp analysis of the log shows one dominant gap:

• 03:53:05.973 — [ RUN ] shm_ib/test_ucp_am_nbx_closed_ep.rx_short_am_on_closed_ep/2 <shm,ib/reply>
• 04:07:43.515 — 14 min 38 s of total silence, then ucp_test.cc:300 UCX ERROR request ... did not complete on time

That gap equals the watchdog_timeout = 900. in test/gtest/common/test_helpers.cc:31, i.e. the watchdog expired and killed the watched thread (test_helpers.cc:57 ADD_FAILURE() << ... "abort testing", then pthread_kill). The backtrace confirms the thread was spinning in __sched_yield inside the progress loop — a live-lock waiting for a completion that never arrives, not a deadlocked mutex.

The last meaningful operation is the send in test_recv_on_closed_ep() (test/gtest/ucp/test_ucp_am.cc:1460), whose request_wait(sreq) at line 1488 never returns; m_recv_counter=0 vs m_send_counter=1 at the EXPECT_EQ on line 593. This test uses UCP_ERR_HANDLING_MODE_PEER and closes the receiver's EP mid-flight (test_ucp_am.cc:1444,1471), so it deliberately drives the UCT error handler → uct_rc_mlx5_ep_outstanding_purge() path — exactly the code the PR HEAD commit modifies.

Two concrete defects in that new code are consistent with a permanent stall:

1. src/uct/ib/mlx5/rc/rc_mlx5_ep.c:1552-1558 — uct_rc_mlx5_ep_outstanding_release_send_op() calls uct_rc_op_release_get_bcopy()/uct_rc_op_release_reads_get_zcopy() + uct_rc_iface_update_reads() for GET ops. uct_rc_iface_update_reads() (rc_iface.h:504) only folds tx.reads_completed into tx.reads_available; if the purge releases the op without first incrementing reads_completed, the RDMA_READ credits are lost permanently. Once tx.reads_available hits zero the iface can never post again and every subsequent send blocks forever — a textbook silent hang.
2. src/uct/ib/mlx5/rc/rc_mlx5_ep.c:1681-1685 — delivered is a loop-carried variable only reassigned when num_packets != 0. Because the new uct_ib_mlx5_wqe_is_delivered() unconditionally returns 0 for MLX5_OPCODE_RDMA_READ (line 1412-1414), a READ WQE poisons delivered for any following zero-packet (NOP) WQE, causing correctly-delivered ops to be cancelled instead of completed — the receiver then never sees the AM (m_recv_counter=0).

Implicated commit: [REDACTED:Hex High Entropy String] — Zihao Zhao, "UCT/IB/MXL5: add get support for outstanding purge" (the PR HEAD; it is the most recent commit touching both src/uct/ib/mlx5/rc/rc_mlx5_ep.c and src/uct/ib/rc/base/rc_ep.h).

File: src/uct/ib/mlx5/rc/rc_mlx5_ep.c:1552 (read-credit release in uct_rc_mlx5_ep_outstanding_release_send_op) and src/uct/ib/mlx5/rc/rc_mlx5_ep.c:1681 (delivered not reset per-WQE)

Suggested fix:

1. In uct_rc_mlx5_ep_outstanding_release_send_op(), increment iface->tx.reads_completed (by the op's length/packet count, as the normal GET completion handler does) before calling uct_rc_iface_update_reads(), so purged GET operations give their RDMA_READ credits back. Add an assertion/ucs_trace on tx.reads_available after purge to catch credit leaks in CI rather than as a 15-minute hang.
2. Reset delivered explicitly at the top of each loop iteration in uct_rc_mlx5_ep_outstanding_purge() (or skip num_packets == 0/NOP WQEs entirely) so a preceding RDMA_READ cannot force a NOP or a genuinely delivered WQE to be reported as cancelled.
3. Since uct_rc_txqp_add_send_comp_always() (src/uct/ib/rc/base/rc_ep.h:418) now allocates an op from iface->tx.free_ops even when comp == NULL, verify every GET completion/purge path returns that op via uct_rc_iface_put_send_op(); a leak here exhausts the free-op list and produces the same hang.
4. Re-run with --gtest_filter='*test_ucp_am_nbx_closed_ep*' and UCX_LOG_LEVEL=trace_poll to confirm which resource (reads_available, cq_available, or free_ops) is pinned at zero.

Related: PR #11866 (this PR, "UCT/IB/MLX5: wqe parser for uct get"); sibling PRs on the same purge series: #11873, #12019, #12006, #11874, #11836. Note test_ucp_am_nbx_send_copy_header.all_protos in the same file is already hard-disabled with the comment /* FIXME: Disabled due to unresolved failure - CI Hang */ (test_ucp_am.cc:1338) — likely the same underlying RC purge hang, so fixing this should be checked against re-enabling that test.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id fbd32f51-5de9-4f05-a2a3-a0e54de60c31 in the triage console for the audit trail.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[evgeny-leksikov]

LGTM besides couple of minor comments

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.