[michal-shalev]

Backport #11947.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

Residual coverage note: offload_proto_selected() matches put/sgl/offload by substring, so it also accepts put/sgl/offload_sw, whose send func uses max_frag_count == 1. The new multi-fragment post path (frag loop building uct_memhs/uct_rkeys) is therefore only really exercised on hardware exposing UCT_IFACE_FLAG_V2_PUT_SGL_ZCOPY (mlx5 plugin / CUDA 13 cuda_ipc); the iterator-level batching is covered by test_ucp_dt_sgl.iter_next_frag_mixed_lengths.

Out of scope here, but worth noting: SGL protocol selection still uses dt_iter->length (the element count) as msg_length, while this PR makes the per-lane split byte-based. The new total length would be the natural input for that too, if it's ever revisited.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (Tests gpu on worker 1) · commit bf7e083f

TL;DR: The failure is in a pure UCT-level test, ud_verbs/uct_p2p_am_misc.am_max_short_iov_multi/2 on device mlx5_1:1 — the UD endpoint's TX window was never acknowledged by the peer, so sends stayed at UCS_ERR_NO_RESOURCE for the full 10 s deadline and the UD slow timer then raised an unhandled UCS_ERR_ENDPOINT_TIMEOUT (ucs_fatal → core dump). This is unrelated to PR #12232 (a UCP RMA SGL-split change); re-run and/or check the mlx5_1 port on swx-rdmz-ucx-gpu-02.

Full analysis

Summary: gtest aborted (make: *** [Makefile:4783: test] Aborted (core dumped)) after ud_verbs/uct_p2p_am_misc.am_max_short_iov_multi/2 <ud_verbs/mlx5_1:1> failed at test/gtest/uct/test_p2p_am.cc:568 (EXPECT_EQ(UCS_OK, status) got -2 = UCS_ERR_NO_RESOURCE), followed by ud_ep.c:293 Fatal: UD endpoint 0x[REDACTED:Hex High Entropy String] to :0: unhandled timeout error.

Root cause: Evidence from the log timeline:

• 15:33:57.227 test starts; the am_max_multi() helper floods the UD ep until resources are exhausted, then loops progress() + send for DEFAULT_TIMEOUT_SEC.
• 15:34:07.256 (exactly ~10 s later) the loop expires with status == UCS_ERR_NO_RESOURCE → the TX window never advanced, i.e. no ACKs ever came back from the peer for the in-flight UD packets.
• 15:34:27.786 (another ~20 s in teardown/flush) the UD slow timer fires uct_ud_ep_deferred_timeout_handler(); uct_iface_handle_ep_err() returns non-OK because these UCT p2p test endpoints are created without a peer-failure error handler, so src/uct/ib/ud/base/ud_ep.c:293 calls ucs_fatal() and kills the whole gtest binary.

The peer name printed as ":0" (empty LID/dest) confirms the UD ep had no reachable/acking peer. Both the test's own send path (uct_ud_verbs_ep_am_short_iov, src/uct/ib/ud/verbs/ud_verbs.c:210) and the timeout handler are entirely in UCT/IB/UD; PR #12232 ("UCP: Split an SGL put between the lanes by bytes - v1.23.x") only touches UCP RMA protocol code and cannot influence a direct UCT UD AM test. So this is an infrastructure/flaky-transport failure on the mlx5_1:1 port of swx-rdmz-ucx-gpu-02, not a regression from the PR. The ucs_fatal is the symptom amplifier: one lost UD window turns a single EXPECT failure into a core dump that fails the whole job.

Implicated commit: unknown — not attributable to [REDACTED:Hex High Entropy String]; the most recent changes in the implicated area are 9aa2c33e (Ilia Yastrebov, "UCT/UD: Remove stale private UD connections -1.23") and 0ee470a8 (Ilia Yastrebov, "UCT/UD: Add generation ID to dest_id to make it unique"), neither part of this PR.

File: src/uct/ib/ud/base/ud_ep.c:293 (fatal) / test/gtest/uct/test_p2p_am.cc:568 (assert) / src/uct/ib/ud/verbs/ud_verbs.c:210 (uct_ud_verbs_ep_am_short_iov)

Suggested fix:

1. Re-trigger the gpu on worker 1 job — this is not a PR regression; verify the mlx5_1 port state (ibstat/link up, LID assigned, port not in Init) on swx-rdmz-ucx-gpu-02, since the UD ep peer printed as ":0" and all other mlx5_1:1 tests in this run skipped.
2. Harden the test so a UD window stall yields a test failure instead of aborting the whole binary: in uct_p2p_am_misc::am_max_multi() create/flush the ep with UCT_IFACE_PARAM_FIELD_ERR_HANDLER / UCT_EP_PARAM_FIELD_ERR_HANDLER (error-handling mode PEER) so uct_iface_handle_ep_err() returns UCS_OK and ud_ep.c:293 can't ucs_fatal; optionally scale the recovery deadline by UCX_UD_TIMER_TICK as no_rx_buffs already does (test_p2p_am.cc:664).
3. If this recurs on the same worker, add the test to the known-flaky list and open a tracking issue referencing the long-standing UD timeout report.

Related: PR #12232 (the PR under test, unrelated to the failure); long-standing UD timeout flakiness issue #5543; prior UD-test hardening precedent #11948

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 191bbaca-05ae-4690-9d55-abec22711836 in the triage console for the audit trail.