UCT/IB/MLX5/RC: Complete SGL op with ep error status on peer failure by dmitrygx · Pull Request #12205 · openucx/ucx

dmitrygx · 2026-10-31T15:18:51Z

What?

With peer error handling, an rc_mlx5 put_sgl_zcopy/get_sgl_zcopy operation completes with UCS_ERR_CANCELED instead of the endpoint error status when the peer fails and the error is reported for any element of the SGL except the last one. This PR keeps the send op of an SGL operation on the first WQE of its chain, so the failure purge completes it like an operation of a single WQE.

Why?

An SGL operation posts one WQE per element and kept its send op and CQ update on the last WQE of the chain. The error CQE is reported for the first failed WQE of the chain, and uct_rc_mlx5_iface_handle_failure() completes the outstanding send ops with sn up to that WQE with the endpoint error status, so the SGL send op was skipped. It was completed later by the flush CQE of its last WQE (WR_FLUSH_ERR) with UCS_ERR_CANCELED, so a user of the HW SGL protocol (UCP put/sgl/offload) could not tell a peer failure from a cancellation.

How?

UCT/IB/MLX5/RC: uct_rc_mlx5_base_ep_sgl_zcopy_post() adds the send op with the sn of the first WQE of the chain instead of the last one. A failure reported for any WQE of the chain is at or after that sn, so the existing purge completes the operation with the endpoint error status. The operation still completes only by the CQE of the last WQE, since the other WQEs of the chain are unsignaled. Operations posted after the failed WQE are still completed by the QP flush, as before. The token based failover purge (uct_rc_mlx5_ep_outstanding_purge()) is not changed; it does not support SGL operations either before or after this change.
GTEST/UCT: add test_uct_peer_failure_sgl_zcopy on rc_mlx5 and cuda_ipc, the transports with SGL zcopy. The tests are skipped by the SGL caps, e.g. on cuda_ipc built with CUDA older than 13; for that uct_test::check_caps() gets optional required v2 caps. On rc_mlx5 the tests:
- fail the first, a middle or the last element of a 4-element put_sgl_zcopy/get_sgl_zcopy and check that the completion status equals the status reported to the error handler;
- fail a zero-length first element (by killing the receiver, since the remote address of a zero-length element is not accessed) and an element after a zero-length one, put only, since get needs elements longer than max_inl_cqe;
- check that an SGL put posted after a failed zero-length put_short, and a put_zcopy posted after a failed SGL put, are completed by the QP flush;
- check that without a failure the operation completes with UCS_OK and moves the data of all the elements but a zero-length one.
On cuda_ipc only the completion tests run, with CUDA memory, since cuda_ipc does not report errors of posted operations. The completion test with a zero-length element is skipped there until UCT/CUDA_IPC: Skip zero-length elements in SGL zcopy #12219, which fixes cuda_ipc failing such an SGL.

The UCP part (SGL operations on failed lanes return the endpoint error status) will come in a separate PR.

Tests (devel builds):

Test commit only, on master (GB200, RoCE): put_fail_first/put_fail_middle/get_fail_first/get_fail_middle fail on rc_mlx5 (UCS_ERR_CANCELED instead of UCS_ERR_CONNECTION_RESET); *_fail_last and put_after_failed_put_short pass.
With the fix (ConnectX, L4 GPU, CUDA 13.3): test_uct_peer_failure_sgl_zcopy 39/39 pass on rc_mlx5 (3 devices); on cuda_ipc put_completion/get_completion pass and the failure tests are skipped.
With the fix, before the zero-length and cuda_ipc tests (GB200, RoCE): test_uct_peer_failure* on rc_mlx5, rc_verbs and dc_mlx5, and rc_mlx5 test_uct_purge_outstanding, test_uct_sgl and test_rc_get_limit: 405/405 pass. test_ucp_rma_sgl: 80/80 pass.

svc-nvidia-pr-review · 2026-10-31T15:18:59Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-10-31T15:33:58Z

+        return pi;
+    }
+
+    if (uct_ib_mlx5_wqe_size(ctrl) > rdma_wqe_size) {


An unsignaled zero-length put_short WQE is also just ctrl+raddr with no inline segment (see the length == 0 branch in uct_rc_mlx5_txqp_inline_post), so it reaches the scan below and can return the sn of an unrelated SGL op posted after it, completing that op before its WQEs are reported. Can we require wqe_size == rdma_wqe_size + sizeof(struct mlx5_wqe_data_seg) instead?

Good catch, thanks. Reproduced by put_after_failed_put_short (7f99f92): without the fix, the SGL put posted after a failed zero-length put_short completed with UCS_ERR_ENDPOINT_TIMEOUT from the put_short failure. Fixed in 57e385e: an SGL chain WQE must now be exactly ctrl + raddr + one data segment, and not inline. A WQE of a zero-length SGL element can't be told from a zero-length put_short, so it is not detected; this is noted in the function comment.

svc-nvidia-pr-review · 2026-10-31T15:33:58Z

+    test_put_sgl_zcopy_peer_failure(SGL_COUNT - 1);
+}
+
+UCT_INSTANTIATE_TEST_CASE(test_uct_peer_failure_sgl_zcopy)


cuda_ipc also reports UCT_IFACE_FLAG_V2_PUT_SGL_ZCOPY (CUDA >= 13) and ERRHANDLE_PEER_FAILURE, but the test allocates host mapped_buffer, which the cuda_ipc MD cannot register (reg_mem_types is CUDA only), so the test would not skip and would fail there.

Suggested change

UCT_INSTANTIATE_TEST_CASE(test_uct_peer_failure_sgl_zcopy)

UCT_INSTANTIATE_RC_TEST_CASE(test_uct_peer_failure_sgl_zcopy)

Alternatively select the memory type from md_attr().reg_mem_types like test_uct_sgl does.

Done in f8e89c1, the test is instantiated with UCT_INSTANTIATE_RC_TEST_CASE.

svc-nvidia-pr-review · 2026-10-31T15:33:58Z

+
+        ASSERT_EQ(1u, comp.done);
+        ASSERT_EQ(1ul, m_failed_eps.count(ep0()));
+        EXPECT_EQ(m_failed_eps[ep0()], comp.uct.status)


Do put_fail_first/put_fail_middle fail without the iface change? The last WQE of the chain is signaled, and its flush-error CQE should purge the op with the same endpoint status, so it would help to state the symptom being fixed in the PR description.

Yes: without the rc_mlx5_iface.c change they complete with UCS_ERR_CANCELED (-16) instead of UCS_ERR_CONNECTION_RESET (-25). The flush-error CQE of the last WQE does not carry the endpoint status: uct_ib_mlx5_completion_with_err() maps WR_FLUSH_ERR to UCS_ERR_CANCELED, and uct_rc_mlx5_iface_handle_failure() purges with the status of that CQE.

svc-nvidia-pr-review · 2026-10-31T15:33:59Z

Residual coverage gap: only put_sgl_zcopy is covered; get_sgl_zcopy shares uct_rc_mlx5_base_ep_sgl_zcopy_post() and the same handler check, but has no peer-failure test.

Residual coverage gap: the UCP rma_sgl variants are end-to-end only — which WQE of the chain fails is nondeterministic, so the middle-of-chain path is covered only by the new UCT tests (i.e. only on rc_mlx5 hardware CI).

svc-ucx · 2026-10-31T16:05:43Z

🤖 CI Triage Agent — UCX PR (AddressSanitizer gpu on worker 3) · commit da13aa5f

TL;DR: The newly added gtest test_uct_peer_failure_sgl_zcopy is instantiated on every transport (including cuda_ipc) but allocates its SGL buffers as host memory, so uct_cuda_ipc_ep_put_sgl_zcopy gets a non-CUDA pointer and trips the fatal ucs_assertv(cuda_device_ordinal >= 0) in cuda_ipc.inl:39 (device ordinal -2 = CU_DEVICE_INVALID), aborting the gtest binary. Restrict the new test case to non-GPU transports (or allocate CUDA-memory buffers) and make uct_cuda_ipc_check_and_push_ctx return an error instead of asserting.

Full analysis

Summary: cuda_ipc/test_uct_peer_failure_sgl_zcopy.put_fail_last/0 <cuda_ipc/cuda> aborted with Assertion 'cuda_device_ordinal >= 0' failed: cuda_device_ordinal=-2, core dumped, make test exited 2 (Bash exit code 2).

Root cause:

Log evidence: the only failing test is the brand-new cuda_ipc/test_uct_peer_failure_sgl_zcopy.put_fail_last/0, with backtrace frame libuct_cuda.so.0(uct_cuda_ipc_ep_put_sgl_zcopy+0x159f) → ucs_fatal_error_format from cuda_ipc.inl:39. Everything before it passes.
The test (test/gtest/uct/test_peer_failure.cc:1597-1689, added by this PR) builds its SGL from mapped_buffer sendbuf(SGL_COUNT * SGL_LENGTH, 0, *m_sender). uct_test::mapped_buffer's mem_type parameter defaults to UCS_MEMORY_TYPE_HOST (test/gtest/uct/uct_test.h:255-259), so buffers[i] are host pointers.
It is registered with UCT_INSTANTIATE_TEST_CASE(test_uct_peer_failure_sgl_zcopy) (line 1689), which includes the CUDA TLs (UCT_TEST_CUDA_MEM_TYPE_TLS → cuda_ipc, uct_test.h:507-510). The skip condition only checks UCT_IFACE_FLAG_ERRHANDLE_PEER_FAILURE plus UCT_IFACE_FLAG_V2_PUT_SGL_ZCOPY/max_put_sgl_zcopy_count, all of which cuda_ipc advertises — so nothing skips it, and there is no check that the local buffer memory type is usable by the TL.
uct_cuda_ipc_post_cuda_sgl_async_copy() calls uct_cuda_ipc_check_and_push_ctx((CUdeviceptr)buffers[0], ...) (src/uct/cuda/cuda_ipc/cuda_ipc_ep.c:321). For a plain host pointer, cuPointerGetAttributes() succeeds but returns CU_POINTER_ATTRIBUTE_DEVICE_ORDINAL = -2 (CU_DEVICE_INVALID) and a NULL context; the unconditional ucs_assertv(cuda_device_ordinal >= 0, ...) at cuda_ipc.inl:39 is therefore fatal in this assertion-enabled AddressSanitizer build.

So this is a new-test/transport-coverage bug exposed by a missing input-validation path in cuda_ipc, not a flake or a timeout.

Implicated commit: da13aa5f — "GTEST/UCT: Check put_sgl_zcopy completion status on peer failure", Dmitrii (the only commit in the PR touching test/gtest/uct/test_peer_failure.cc); the asserting code is pre-existing (cuda_ipc.inl introduced with the cuda_ipc SGL work, e.g. [REDACTED:Hex High Entropy String]/9466d104, Michal Shalev).

File: test/gtest/uct/test_peer_failure.cc:1624 (host mapped_buffer) and :1689 (UCT_INSTANTIATE_TEST_CASE); fatal assert at src/uct/cuda/cuda_ipc/cuda_ipc.inl:39, reached from src/uct/cuda/cuda_ipc/cuda_ipc_ep.c:321.

Suggested fix:

In the test, stop instantiating the new case on GPU transports, e.g. replace
UCT_INSTANTIATE_TEST_CASE(test_uct_peer_failure_sgl_zcopy) with
UCT_INSTANTIATE_NO_GPU_TEST_CASE(test_uct_peer_failure_sgl_zcopy) (or _UCT_INSTANTIATE_TEST_CASE(..., rc_mlx5) / UCT_INSTANTIATE_RC_DC_TEST_CASE, since the test targets IB SGL peer-failure semantics). Alternatively allocate the buffers with the memory type the TL actually supports (pass UCS_MEMORY_TYPE_CUDA for cuda TLs) and add a skip when md_attr doesn't list host memory in access_mem_types.
Harden the transport so a bad pointer can never abort a release/debug run: in uct_cuda_ipc_check_and_push_ctx(), replace the ucs_assertv with a graceful error, e.g.
```
if (ucs_unlikely(cuda_device_ordinal < 0)) {
    ucs_debug("non-cuda address %p (device_ordinal=%d)", (void*)address,
              cuda_device_ordinal);
    return UCS_ERR_INVALID_PARAM;
}
```
so uct_ep_put_sgl_zcopy returns an error instead of killing the process.

Related: PR #12205 ("UCP/UCT/IB: Return endpoint error status for SGL put on peer failure") — #12205; cuda_ipc SGL implementation PRs #11359 and #11762.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id bcf300c8-54c9-46b5-a1fe-c3cda55d1110 in the triage console for the audit trail.

svc-nvidia-pr-review · 2026-11-01T13:14:20Z

🤖 Starting review — findings will be posted here when done.

svc-ucx · 2026-11-01T13:24:36Z

🤖 CI Triage Agent — UCX PR (Tests roce on worker 0) · commit 3637358a

TL;DR: The build was not a code failure — the Azure DevOps agent on agent-03 received a shutdown signal while still compiling gtest sources, so the job was cancelled mid-build; re-run the pipeline (and check the agent host's health/service state).

Full analysis

Summary: Job "Tests roce on worker 0" (build 141904) was aborted during the Run ./contrib/test_jenkins.sh step while compiling test/gtest/** — no test ever ran.

Root cause: Infrastructure, not code. The log shows a clean build with zero compiler/linker errors right up to the end (libucs/libuct/libucp, tools, examples and gtest objects all built fine). At 13:23:00.6320019Z the log contains ##[error]The agent has received a shutdown signal. This can happen when the agent service is stopped, or a manually started agent is canceled., immediately followed by ##[error]The Operation will be canceled and ##[error]The operation was canceled, then Azure's orphan-process cleanup killing the in-flight make/g++/cc1plus processes on [REDACTED:Base64 High Entropy String].

This is not a hang or a wall-clock timeout: timestamps are continuous with gaps of only 1–6 seconds (e.g. 13:22:54 → 13:22:55 → 13:22:59 → 13:23:00) all the way to the cancellation, and the step had only been running ~3 minutes of compilation. The work was progressing normally when the agent process went away.

Implicated commit: unknown — no commit is implicated; the aborted step produced no error attributable to [REDACTED:Hex High Entropy String] (PR #12205, topic/ucx/sgl-peer-failure). Nothing from that branch was even exercised (compilation of uct/test_peer_failure.cc completed successfully at 13:22:51).

File: n/a — failure originates in the Azure Pipelines agent on host agent-03, workspace [REDACTED:Base64 High Entropy String]

Suggested fix:

Simply re-run / re-queue the roce on worker 0 job for this PR — this failure carries no signal about the code change.
On the CI side, have an admin check agent-03: whether the vsts-agent systemd service was restarted, the host was rebooted/drained, or the agent was manually cancelled around 2026-11-01 13:23 UTC. Recurrent shutdown signals on the same agent usually mean an unattended agent auto-update, an OOM/reboot, or a maintenance job stopping the service while builds are live.
If agent-03 is flaky, take it out of the ROCE pool (similar in spirit to PR CI: Run the new and GPU jobs on new-01 only (test, do not merge) #12194, which pins new/GPU jobs to a specific worker) until the agent service is stable, so PRs don't get spurious red runs.

Related: #12194 (CI: pinning jobs to specific workers) — tangentially related CI-infrastructure work; no existing issue tracks this agent shutdown.

🛡️ This comment had 2 potential secret(s) redacted (Base64 High Entropy String, Hex High Entropy String). See request_id d9fab973-6093-42cb-9bcc-e3583d712cea in the triage console for the audit trail.

svc-nvidia-pr-review · 2026-11-01T13:28:24Z

src/uct/ib/mlx5/rc/rc_mlx5_iface.c

maybe we can identify the SGL chain from the send op instead of parsing the WQE, e.g. keep the chain's first sn (or WQE count) in the op? these sizes duplicate the layout built by uct_rc_mlx5_base_ep_sgl_zcopy_post(), so the detection silently stops working if that layout changes, and it would also cover the zero-length element case the comment says cannot be detected.

src/ucp/core/ucp_ep.c

can we copy the stub's current internal_ops and override only ep_put_sgl_zcopy/ep_get_sgl_zcopy, instead of duplicating the whole table? otherwise any internal op added later in UCT stays NULL here and will crash when called, and the other entries change failed-lane behavior beyond this fix.

Residual coverage note: the err_handler_inprogress path (failover with UCT_IFACE_FLAG_V2_QUERY_TOKEN) still defers to uct_rc_mlx5_ep_outstanding_purge(), whose WQE walk maps WQEs 1:1 to ops via uct_rc_mlx5_ep_outstanding_peek_send_op() and would report each SGL element as a separate put_zcopy. That is pre-existing and untested for SGL, so out of scope here, but worth knowing the fix only covers the non-failover purge path.

svc-nvidia-pr-review · 2026-11-01T17:26:35Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-11-01T17:42:10Z

+}
+
+/* SGL put protocols use only RMA_BW lanes which support put zcopy */
+bool test_ucp_peer_failure::has_put_zcopy_rma_bw_lane()


minor: can we use sender().has_lane_with_caps(UCT_IFACE_FLAG_PUT_ZCOPY) here instead of a new helper? test_ucp_rma_sgl uses it for the same skip, and it also accepts v2 caps if we want to key the skip on UCT_IFACE_FLAG_V2_PUT_SGL_ZCOPY.

Impact: ~20 lines of duplicated lane/iface-attr walking in a fixture, against test/AGENTS.md ("Reuse helpers from test/gtest/common and existing fixtures before adding a new fixture shape").

This is what the test used first, but it is too weak: on tcp a non-RMA lane has PUT_ZCOPY while no put/sgl protocol is selected ("cannot find remote protocol"), so the test failed instead of skipping. Keying on UCT_IFACE_FLAG_V2_PUT_SGL_ZCOPY would skip rc, dcx and dc_mlx5, where put/sgl/offload_sw is used and the test passes. The helper checks the RMA_BW lanes, which is what both SGL put protocols use.

dmitrygx · 2026-11-01T18:04:15Z

get_sgl_zcopy is covered now by get_fail_first/get_fail_middle/get_fail_last (f8e89c1).

dmitrygx · 2026-11-01T18:12:22Z

src/uct/ib/mlx5/rc/rc_mlx5_iface.c: maybe we can identify the SGL chain from the send op instead of parsing the WQE

Considered it, but uct_rc_iface_send_op_t has no spare field: length is used to release RDMA READ credits of get ops and the union holds iface, so keeping the chain start there would grow every send op on the fast path. The WQE layout check matches uct_rc_mlx5_base_ep_sgl_zcopy_post() (same non_data_wqe_size/data_wqe_size names), and a zero-length element keeps the pre-existing behavior (completed by the QP flush with UCS_ERR_CANCELED).

src/ucp/core/ucp_ep.c: can we copy the stub's current internal_ops and override only ep_put_sgl_zcopy/ep_get_sgl_zcopy

The table follows ucp_failed_tl_iface_ops and sets all 12 fields of uct_iface_internal_ops_t, like the internal ops table of every UCT iface, so a new internal op has to be added here the same way as to the public ops. Returning the endpoint failure status from all endpoint ops of a failed lane, instead of the UCT stub's UCS_ERR_CANCELED, is intended and matches the public ops of ucp_failed_tl_iface.

dmitrygx · 2026-11-01T18:14:49Z

@michal-shalev @evgeny-leksikov could you review these chages, please?

evgeny-leksikov

please split the pull request to independent parts and clarify exactly what is each part needed for. For example purge outstanding is not needed for per_failure err handling. It's needed for token based failover and recovery.

svc-nvidia-pr-review · 2026-11-02T17:29:21Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-11-02T17:40:19Z

                                   uint16_t op_flags, uct_completion_t *comp)
 {
    uct_ib_mlx5_txwq_t *txwq       = &ep->tx.wq;
+    const uint16_t first_pi        = txwq->sw_pi;


minor: first_pi and pi are both initialized from the same txwq->sw_pi (pi = txwq->sw_pi; a few lines below), i.e. redundant state that can silently drift. Can we set pi = first_pi there so the two cannot drift apart?

svc-nvidia-pr-review · 2026-11-02T17:40:21Z

Residual coverage note: the new tests only exercise the non-failover purge path (test_uct_peer_failure::err_cb returns UCS_OK). The err_handler_inprogress path (UCT_IFACE_FLAG_V2_QUERY_TOKEN) still goes through uct_rc_mlx5_ep_outstanding_purge(), whose WQE walk maps one WQE to one op (uct_rc_mlx5_ep_outstanding_peek_send_op()); the SGL handlers are not recognized by uct_rc_mlx5_op_info_fill_put/get(). This was already raised previously as pre-existing and out of scope — with this change the failing WQE shifts from the last to the first of the chain, but the outcome is the same, so not re-opened.

test_sgl_zcopy_after_failed_put_short depends on the zero-length put_short WQE being the one reported first (retry-exceeded on the oldest unacked WQE). This matches the existing test_uct_peer_failure.peer_failure pattern, so it should be stable, but it is an ordering assumption rather than an enforced one.

The suite is instantiated via UCT_INSTANTIATE_RC_TEST_CASE, so rc_verbs always skips (only rc_mlx5 implements SGL zcopy); effective coverage is rc_mlx5 hardware CI only.

svc-nvidia-pr-review · 2026-11-02T20:12:46Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-11-02T20:23:17Z

🤖 Review complete — no issues found.

svc-nvidia-pr-review · 2026-11-02T20:23:58Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-11-02T20:33:21Z

🤖 Review complete — no issues found.

svc-nvidia-pr-review · 2026-11-02T20:34:49Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-11-02T20:45:37Z

+
+    /* Without a failure the operation completes with UCS_OK by the CQE of the
+     * last WQE of the chain, so the data of all its elements is in place */
+    void test_sgl_zcopy_completion(bool is_put_op)


maybe move these success-path checks to test_uct_sgl, which already has with_callback? They don't inject any failure, and this fixture lowers RC_TIMEOUT to 100us / RC_RETRY_COUNT to 4, which adds flakiness risk to a test that asserts m_failed_eps.count(ep0()) == 0.

Rationale: put_completion flushes before pattern_check(), so it cannot detect a premature completion and is effectively a duplicate of test_uct_sgl.with_callback; only get_completion (data checked without flush) adds a new assertion.

svc-nvidia-pr-review · 2026-11-02T20:46:19Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-11-02T20:57:52Z

+    test_put_zcopy_after_failed_sgl_zcopy();
+}
+
+UCS_TEST_SKIP_COND_P(test_uct_peer_failure_sgl_zcopy, put_completion,


minor: maybe drop put_completion/get_completion? test_uct_sgl.with_callback in test/gtest/uct/v2/test_uct_sgl.cc already covers the multi-element success path for put and get (completion fired with UCS_OK + pattern_check after flush) on all transports, so this duplicates it in a peer-failure fixture.

svc-nvidia-pr-review · 2026-11-03T10:39:15Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-11-03T10:48:00Z

🤖 Review complete — no issues found.

jeynmann · 2026-11-03T11:04:49Z

+                     !check_caps(UCT_IFACE_FLAG_ERRHANDLE_PEER_FAILURE, 0,
+                                 UCT_IFACE_FLAG_V2_PUT_SGL_ZCOPY))
+{
+    test_sgl_zcopy_completion(true);


Do we need add case for success path in test_uct_peer_failure_sgl_zcopy ?

svc-nvidia-pr-review · 2026-11-03T11:10:45Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-11-03T11:21:32Z


    bool is_caps_supported(uint64_t required_flags);
-    bool check_caps(uint64_t required_flags, uint64_t invalid_flags = 0);
+    bool check_caps(uint64_t required_flags, uint64_t invalid_flags = 0,


minor: can we keep the existing idiom !check_caps(A) || !check_caps_v2(B) (as in test_rc.cc) instead of adding a third positional parameter? the 0 for invalid_flags repeated in all 10 new skip conditions is less readable than two calls.

svc-ucx · 2026-11-03T14:05:21Z

🤖 CI Triage Agent — UCX PR (Tests new on worker 3) · commit ba9cb210

TL;DR: The gtest suite on worker 3 reported exactly one failure — tcp/test_ucp_rma.put_nonblocking/0 <tcp,cuda_copy,rocm_copy/flush_worker> — in the TCP software-emulated RMA put path, which PR #12205 (UCT/IB/MLX5/RC SGL peer-failure completion) does not touch; this looks like an unrelated/flaky data-validation failure, so re-run the job and capture the full gtest output before treating it as a regression.

Full analysis

Summary: make test exited with code 2 because 1 of 8903 gtests failed: tcp/test_ucp_rma.put_nonblocking/0, where GetParam() = tcp,cuda_copy,rocm_copy/flush_worker.

Root cause: From the log, the run completed normally (8902 passed, 3361 skipped, 3,122,763 ms total, continuous timestamped activity right up to the tear-down at 14:02:32 — no gaps, no timeout, no crash/signal, no sanitizer or assertion abort). The only failing case is the non-blocking PUT test over the tcp transport with the flush_worker variant. That path is UCP's software/AM-emulated RMA put plus ucp_worker_flush, followed by buffer comparison in test_ucp_memheap::test_xfer(); the failure is almost certainly the ADD_FAILURE() << "data validation failed" branch there (the per-case assertion text is above the 500-line tail the log API returns, so this specific line is inferred, not quoted).

Crucially, the change under test (PR #12205, "UCT/IB/MLX5/RC: Complete SGL op with ep error status on peer failure", branch topic/ucx/sgl-peer-failure, commit ba9cb21) only affects SGL operations on the IB mlx5/RC transports. The failing test uses tcp and a plain contiguous ucp_put_nbx — it never reaches the mlx5 SGL completion code, and all rc_x/dc_x/rc_mlx5 RMA and SGL cases in the same run passed. So there is no evidentiary link between the failure and this PR; the suspect areas for a real regression in this path would be recent generic flush work, e.g. d513f98 "UCP/FLUSH: Add topology-safe selective endpoint flush (#11923)" (nbellalou) and [REDACTED:Hex High Entropy String] "UCP/PROTO: Repost GET and zcopy on the selected lane (#12091)" (Evgeny Leksikov), both of which touch src/ucp/rma/flush.c.

Implicated commit: unknown — not attributable to [REDACTED:Hex High Entropy String] from the log evidence (PR touches mlx5/RC SGL only; failure is in the tcp non-SGL put path)

File: test/gtest/ucp/test_ucp_memheap.cc:110-116 (validation point reached from test/gtest/ucp/test_ucp_rma.cc:283 put_nonblocking → put_nbi)

Suggested fix:

Re-run the Azure job to confirm; if it passes, label it flaky and do not block the PR.
If it reproduces, get the full assertion text (it was truncated from the log tail) and then reproduce locally with the single case in a loop:
./test/gtest/gtest --gtest_filter='tcp/test_ucp_rma.put_nonblocking/0' --gtest_repeat=100 --gtest_break_on_failure with UCX_LOG_LEVEL=info UCX_PROTO_INFO=y to see which put protocol/lane split was selected (the test forces MAX_RMA_LANES=2).
If it is a genuine regression, focus on worker-flush completeness for emulated (AM-based) PUT over TCP — i.e. whether ucp_worker_flush guarantees remote completion of all fire-and-forget put_nbi requests after the selective-flush rework in src/ucp/rma/flush.c (UCP/FLUSH: Add topology-safe selective endpoint flush #11923) — not on the SGL/mlx5 code in this PR.
Consider making the test print the failing offset/expected/actual in test_ucp_memheap::test_xfer() so truncated CI logs still identify the corruption pattern.

Related: PR #12205 (the PR under test), #11923 (UCP/FLUSH selective endpoint flush), #12091 (repost GET/zcopy on selected lane)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 4e6ccec2-ab8e-4cd6-b856-9271cbb421ad in the triage console for the audit trail.

dmitrygx · 2026-11-03T15:08:08Z

/azp run

azure-pipelines · 2026-11-03T15:09:34Z

Azure Pipelines successfully started running 4 pipeline(s).

svc-nvidia-pr-review · 2026-11-03T19:13:32Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-11-03T19:23:53Z

🤖 Review complete — no issues found.

An SGL operation posts one WQE per element, and only the last WQE is signaled. Its send op was kept on the last WQE, so the peer failure purge, which completes the operations up to the failed WQE with the endpoint error status, missed it when an element other than the last one failed, and the operation was completed later by the QP flush with UCS_ERR_CANCELED. Keep the send op on the first WQE of the chain, so a failure of any of its WQEs completes the operation with the endpoint error status, like an operation of a single WQE. Without a failure it is still completed by the CQE of the last WQE. Add test_uct_peer_failure_sgl_zcopy on rc_mlx5 and cuda_ipc: a failure of the first, a middle or the last element of an SGL put/get, SGL puts with a zero-length element, an SGL put posted after a failed put_short, a put_zcopy posted after a failed SGL put, and completion without a failure. uct_test::check_caps() gets optional required v2 caps for the SGL caps check.

svc-nvidia-pr-review · 2026-11-03T19:52:58Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-11-03T20:05:42Z

Residual note (not blocking): test_sgl_zcopy_after_failed_put_short relies on the zero-length put_short WQE being the one reported first (retry-exceeded on the oldest unacked WQE). This is the same assumption as the existing test_uct_peer_failure.peer_failure, so it should be stable, but it is an ordering assumption rather than an enforced one.

Residual note (not blocking): the failover purge path (err_handler_inprogress, UCT_IFACE_FLAG_V2_QUERY_TOKEN) remains SGL-unaware: uct_rc_mlx5_op_info_fill_put/get() has no branch for the SGL handlers, so an undelivered chain WQE still hits ucs_fatal("unexpected put wqe send op") (rc_mlx5_ep.c:1514). With this change the op is matched at the chain's first WQE instead of its last, adding one new sub-case (first WQE delivered, a later one not -> user completion fires with UCS_OK while the later WQEs are reported for retransmission). Previously raised as pre-existing/out of scope.

Residual note (not blocking): test_uct_peer_failure_sgl_zcopy only covers the non-failover purge path, since test_uct_peer_failure::err_cb() returns UCS_OK.

Minor judgement call (not blocking): test_sgl_zcopy_empty_element_failure() is nearly a copy of test_sgl_zcopy_peer_failure() (differs only by killing the receiver and zeroing the remote addresses); the two could share one helper.

svc-ucx · 2026-11-03T21:17:56Z

🤖 CI Triage Agent — UCX PR (Build build_aarch64 rocky96_aarch64) · commit 50be3cf9

TL;DR: The job never built anything — it died in the "Get sources" step because git fetch had no usable GitHub credentials ("could not read Username for 'https://github.com': terminal prompts disabled") through all 5 retries. This is a CI infrastructure/auth problem, not a code problem in PR #12205; re-run the build and, if it persists, refresh the GitHub service-connection credential used by the aarch64 agent pool.

Full analysis

Summary: Azure Pipelines job build_aarch64 rocky96_aarch64 (build 142522) failed in the Checkout openucx/ucx@refs/pull/12205/merge to s task with Git fetch failed with exit code: 128.

Root cause: Authentication failure at source checkout. The agent (ucx-ci-giga01-11) initialized the container successfully, then ran:

git --config-env=http.extraheader=env_var_http.extraheader fetch --force --tags --prune ... origin --depth=100 +[REDACTED:Hex High Entropy String]:refs/remotes/origin/[REDACTED:Hex High Entropy String]...
fatal: could not read Username for 'https://github.com': terminal prompts disabled

Every attempt (3 inner retries × 5 outer RetryHelper attempts, 21:13:43 → 21:16:01) failed identically and instantly (~250 ms), i.e. git fell straight through to an interactive credential prompt rather than being rejected by the server. Immediately before the fetch, git config --get-all http.extraheader and git config --get-regexp .*extraheader both returned nothing, which indicates the credential the agent was supposed to inject via the env_var_http.extraheader environment variable was empty/missing (expired or unauthorized GitHub service connection / PAT, or a token-injection failure on this agent).

There is zero build or compile output in the log — no configure, no make, no test ran — so nothing in the branch content ([REDACTED:Hex High Entropy String], topic/ucx/sgl-peer-failure) can have contributed. The follow-on noise (fatal: ambiguous argument 'HEAD', "delete source folder instead") is just the retry logic cleaning an empty repo, and "Skipping step due to condition evaluation: SucceededNode() → False" shows all subsequent build steps were skipped.

Implicated commit: unknown — infrastructure failure, unrelated to the PR commit [REDACTED:Hex High Entropy String]

File: n/a (failure in Azure Pipelines "Get sources" task, before any repo file was fetched; no .azure/*.yml step reached)

Suggested fix:

Simply re-run the failed job — transient token-injection failures on a single agent are the most common cause and usually clear on retry.
If it reproduces, have CI admins check/renew the GitHub service connection credential for the openucx/ucx pipeline (re-authorize the GitHub App or rotate the PAT; rotate the old one once replaced) and confirm "Allow scripts to access the OAuth token"/endpoint auth is granted for the aarch64 agent pool.
Check agent ucx-ci-giga01-11 specifically — if other agents in the same pool are checking out fine, quarantine this one and verify its agent config/credential cache (agent version 3.246.0, workspace /scrap/azure/agent-11).
Optional hardening: the fetch requests a bare SHA ref (+[REDACTED:Hex High Entropy String]...:refs/remotes/origin/94ce7d05...) with --depth=100; once auth is restored, confirm that still resolves, since merge-ref SHAs can be garbage-collected if the PR is updated while the build is queued.

Related: none found — a search for this error signature in the repo's issues/PRs returned no matching reports (only unrelated PRs).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id a93198a3-c179-4b50-9c81-b903eed4eff2 in the triage console for the audit trail.

dmitrygx · 2026-11-04T10:29:24Z

@evgeny-leksikov @jeynmann @michal-shalev could you approve the PR again? the commits were rebased and squashed

dmitrygx · 2026-11-04T12:19:34Z

/azp run

azure-pipelines · 2026-11-04T12:20:01Z

Azure Pipelines successfully started running 4 pipeline(s).

svc-ucx · 2026-11-04T19:43:24Z

🤖 CI Triage Agent — UCX PR (Tests new on worker 0) · commit 50be3cf9

TL;DR: The gtest binary aborted on ucs_assert(!(region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE)) (src/ucs/memory/rcache.c:396) while unmapping the memheap in tcp/test_ucp_rma.put_nonblocking/2 <...user_memh> — the last reference of a UCP rcache region was dropped while the region was still linked in the page table (an unbalanced ucp_memh_put on the parent memh, destroyed in place by the "unsafe" put fast path). This is in UCP/UCS memory-registration code that PR #12205 (UCT/IB/MLX5/RC SGL completion) does not touch, so it is a pre-existing refcount bug/flake, not a PR regression.

Full analysis

Summary: make test aborted (core dumped) in the gtest stage: assertion failure in ucs_mem_region_destroy_internal() during ucp_mem_unmap() of the receiver memheap in tcp/test_ucp_rma.put_nonblocking/2 <tcp,cuda_copy,rocm_copy/user_memh>.

Root cause: Backtrace from the log: ucp_mem_unmap → ucp_memh_cleanup (ucp_mm.c:470) → ucp_memh_put_rcache (ucp_mm.c:441) → ucs_rcache_region_put_unsafe (rcache.inl:128) → ucs_mem_region_destroy_internal (rcache.c:396). A region inserted in the page table always owns one reference (region->refcount = 2; /* Page-table + user */, rcache.c:1028), so reaching refcount 0 in this put means one reference too many was released on the parent memh (unbalanced put / lost ucs_rcache_region_hold). The failure is then turned into a hard abort by ucs_rcache_region_put_unsafe(), which decrements the refcount non-atomically (unlike every other rcache path, which uses ucs_atomic_fsub32 under pgt_lock) and, on reaching zero, calls ucs_mem_region_destroy_internal() directly without removing the region from the page table and without the write lock — i.e. in a release build this would silently free a region still reachable from the pgtable (use-after-free). The failing test uses TCP + host memory only (no IB RC, no SGL path), and the 19:36:28 → 19:38:12 log gaps are just UCX backtrace/addr2line generation, not a hang.

Implicated commit: unknown (not [REDACTED:Hex High Entropy String] / PR #12205 — it only touches UCT/IB/MLX5/RC). The unsafe lock-free put path originates in 37c574df (Liu, Changcheng, "UCP/MM: lock free when deregistration memory region") and 0ed92911 (Artemy Kovalyov, "UCP/MM: Remove rcache locking"); recent nearby churn: b6b85316 (tomerg-nvidia, mem-registration flags) and d65ffe7b (Michal Shalev, SGL datatype, touched ucp_mm.inl).

File: src/ucs/memory/rcache.inl:121-132 (ucs_rcache_region_put_unsafe), triggering src/ucs/memory/rcache.c:396; caller src/ucp/core/ucp_mm.c:441 (ucp_memh_put_rcache)

Suggested fix:

Re-run the job / run the same test on master to confirm PR UCT/IB/MLX5/RC: Complete SGL op with ep error status on peer failure #12205 is not the cause (it almost certainly isn't) and unblock the PR.
Fix the refcount handling: in ucs_rcache_region_put_unsafe() use ucs_atomic_fsub32() like ucs_rcache_region_put_internal(), and never destroy a region in place — if the last reference drops while UCS_RCACHE_REGION_FLAG_PGTABLE is set, go through ucs_rcache_region_put()/ucs_rcache_region_invalidate_internal() (which takes pgt_lock in write mode and calls ucs_pgtable_remove()), or assert at the put site: ucs_assertv(region->refcount > 1 || !(region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE), ...) so the extra put is caught where it happens rather than at destroy time.
To locate the unbalanced release, re-run gtest --gtest_filter='tcp/test_ucp_rma.put_nonblocking*' with UCX_LOG_LEVEL=trace and inspect the hold / put region traces (ucs_rcache_region_trace) for the memheap region; prime suspects are the invalidate-then-put sequence in ucp_memh_find_slow() (src/ucp/core/ucp_mm.c:1031-1034) and the err_put_rcache paths (src/ucp/core/ucp_mm.c:970, 1091).

Related: PR under test: #12205 (unrelated to the failure). No existing issue found for this assertion; worth opening one for the ucs_rcache_region_put_unsafe destroy-in-pgtable hazard.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 32170dc6-2387-4d9e-be80-0be2fa2bf87e in the triage console for the audit trail.

dmitrygx · 2026-11-05T09:52:01Z

/azp run

azure-pipelines · 2026-11-05T09:52:32Z

Azure Pipelines successfully started running 4 pipeline(s).

svc-nvidia-pr-review reviewed Oct 31, 2026

View reviewed changes

dmitrygx requested a review from a team as a code owner November 1, 2026 13:13

dmitrygx force-pushed the topic/ucx/sgl-peer-failure branch from 3637358 to 57e385e Compare November 1, 2026 17:26

svc-nvidia-pr-review reviewed Nov 1, 2026

View reviewed changes

evgeny-leksikov requested changes Nov 2, 2026

View reviewed changes

dmitrygx force-pushed the topic/ucx/sgl-peer-failure branch from 57e385e to 85f2685 Compare November 2, 2026 17:28

dmitrygx changed the title ~~UCP/UCT/IB: Return endpoint error status for SGL put on peer failure~~ UCT/IB/MLX5/RC: Complete SGL op with ep error status on peer failure Nov 2, 2026

svc-nvidia-pr-review reviewed Nov 2, 2026

View reviewed changes

evgeny-leksikov reviewed Nov 2, 2026

View reviewed changes

Comment thread test/gtest/uct/test_peer_failure.cc Resolved

Comment thread test/gtest/uct/test_peer_failure.cc Resolved

Comment thread test/gtest/uct/test_peer_failure.cc Resolved

svc-nvidia-pr-review reviewed Nov 2, 2026

View reviewed changes

evgeny-leksikov reviewed Nov 2, 2026

View reviewed changes

Comment thread test/gtest/uct/test_peer_failure.cc Resolved

svc-nvidia-pr-review reviewed Nov 2, 2026

View reviewed changes

jeynmann reviewed Nov 3, 2026

View reviewed changes

svc-nvidia-pr-review reviewed Nov 3, 2026

View reviewed changes

jeynmann approved these changes Nov 3, 2026

View reviewed changes

michal-shalev approved these changes Nov 3, 2026

View reviewed changes

dmitrygx enabled auto-merge (squash) November 3, 2026 13:13

evgeny-leksikov approved these changes Nov 3, 2026

View reviewed changes

dmitrygx force-pushed the topic/ucx/sgl-peer-failure branch from ba9cb21 to 2cea8f4 Compare November 3, 2026 19:13

dmitrygx force-pushed the topic/ucx/sgl-peer-failure branch from 2cea8f4 to 50be3cf Compare November 3, 2026 19:52

svc-nvidia-pr-review reviewed Nov 3, 2026

View reviewed changes

Comment thread test/gtest/uct/uct_test.h Resolved

dmitrygx disabled auto-merge November 4, 2026 15:01

dmitrygx enabled auto-merge (squash) November 4, 2026 15:01

dmitrygx mentioned this pull request Nov 5, 2026

UCP/FLUSH: Flush the other lanes after the pending lane #12266

Open

	UCT_INSTANTIATE_TEST_CASE(test_uct_peer_failure_sgl_zcopy)
	UCT_INSTANTIATE_RC_TEST_CASE(test_uct_peer_failure_sgl_zcopy)

Conversation

dmitrygx commented Oct 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

Why?

How?

Uh oh!

svc-nvidia-pr-review commented Oct 31, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svc-nvidia-pr-review commented Oct 31, 2026

Uh oh!

svc-ucx commented Oct 31, 2026

Uh oh!

svc-nvidia-pr-review commented Nov 1, 2026

Uh oh!

svc-ucx commented Nov 1, 2026

Uh oh!

svc-nvidia-pr-review commented Nov 1, 2026

Uh oh!

svc-nvidia-pr-review commented Nov 1, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmitrygx commented Nov 1, 2026

Uh oh!

dmitrygx commented Nov 1, 2026

Uh oh!

dmitrygx commented Nov 1, 2026

Uh oh!

evgeny-leksikov left a comment

Choose a reason for hiding this comment

Uh oh!

svc-nvidia-pr-review commented Nov 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svc-nvidia-pr-review commented Nov 2, 2026

Uh oh!

svc-nvidia-pr-review commented Nov 2, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

svc-nvidia-pr-review commented Nov 2, 2026

Uh oh!

svc-nvidia-pr-review commented Nov 2, 2026

Uh oh!

svc-nvidia-pr-review commented Nov 2, 2026

Uh oh!

svc-nvidia-pr-review commented Nov 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svc-nvidia-pr-review commented Nov 2, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svc-nvidia-pr-review commented Nov 3, 2026

Uh oh!

svc-nvidia-pr-review commented Nov 3, 2026

Uh oh!

jeynmann Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmitrygx commented Oct 31, 2026 •

edited

Loading

jeynmann Nov 3, 2026 •

edited

Loading