[usberkeley]

What?

Use the total SGL byte length for PUT-SGL protocol selection and reconfiguration. Keep the CUDA SGL tests skipped for now.

Why?

The SGL iterator length is an element count, but protocol selection treated it as bytes. For example, four 2 KiB entries were selected as a 4-byte message instead of 8 KiB. This could lead to incorrect protocol ranges and reconfiguration decisions.

The sporadic CUDA failure reproduced in this PR has a separate root cause in cuda_copy reachability and is
outside the scope of this fix.

How?

Calculate the total byte length once, with overflow checking, and store it in the existing request length field. Use that value for initial selection, reconfiguration, and error reporting. Add focused regression tests for the byte length and overflow cases.

Leave the CUDA SGL tests skipped. They will be re-enabled in a separate change that fixes the cuda_copy reachability issue.

[brminich]

@usberkeley thanks for the patch! Do you have CLA signed?

[usberkeley]

@usberkeley thanks for the patch! Do you have CLA signed?

Hi @brminich
I’m not sure. How can I check whether I’ve signed it? This is my first PR for UCX

Just to clarify, I’ve added my name to the AUTHORS file – I’m not sure if that’s what you’re referring to.

[michal-shalev]

@usberkeley thanks for the patch! Do you have CLA signed?

Hi @brminich I’m not sure. How can I check whether I’ve signed it? This is my first PR for UCX

Just to clarify, I’ve added my name to the AUTHORS file – I’m not sure if that’s what you’re referring to.

According to the Guidance for contributors:

UCX uses Git for source code management, Azure CI for continuous integration, and Google’s C++ test framework for unit testing. Before you make any contributions to the projects, please sign a copy of our CLA https://openucx.org/license/ and email the CLA to admin at ucfconsortium.org

you need to sign the UCX CLA and send it to admin@ucfconsortium.org.

[usberkeley]

@usberkeley thanks for the patch! Do you have CLA signed?

Hi @brminich I’m not sure. How can I check whether I’ve signed it? This is my first PR for UCX
Just to clarify, I’ve added my name to the AUTHORS file – I’m not sure if that’s what you’re referring to.

According to the Guidance for contributors:

UCX uses Git for source code management, Azure CI for continuous integration, and Google’s C++ test framework for unit testing. Before you make any contributions to the projects, please sign a copy of our CLA https://openucx.org/license/ and email the CLA to admin at ucfconsortium.org

you need to sign the UCX CLA and send it to admin@ucfconsortium.org.

Thanks for the detailed guidance — Done, I’ve sent the CLA email

[usberkeley]

Hi @michal-shalev
I found that the incorrect data size calculation is not the cause of the “CUDA SGL transport protocol not found” error.

This PR will first fix the data size issue. Would it make sense to re-enable the skipped CUDA test first, and fix the actual root cause in a separate PR?

Root cause analysis

The test uses two UCP workers in the same process. cuda_ipc rejects same-process communication, so it should fall back to cuda_copy.

However, each cuda_copy iface gets a different random ID (cuda_copy_iface.c:341), while connection setup requires the IDs to match (cuda_copy_iface.c:89). As a result, the two workers are incorrectly considered unreachable.

Also, although the CI driver supports CUDA 13.0, UCX is built with CUDA 12.8. CUDA IPC only advertises native SGL support when built with CUDA >= 13.0.

[michal-shalev]

Hi @michal-shalev I found that the incorrect data size calculation is not the cause of the “CUDA SGL transport protocol not found” error.

This PR will first fix the data size issue. Would it make sense to re-enable the skipped CUDA test first, and fix the actual root cause in a separate PR?

Root cause analysis

The test uses two UCP workers in the same process. cuda_ipc rejects same-process communication, so it should fall back to cuda_copy.

However, each cuda_copy iface gets a different random ID (cuda_copy_iface.c:341), while connection setup requires the IDs to match (cuda_copy_iface.c:89). As a result, the two workers are incorrectly considered unreachable.

Also, although the CI driver supports CUDA 13.0, UCX is built with CUDA 12.8. CUDA IPC only advertises native SGL support when built with CUDA >= 13.0.

Thanks for digging into this. I can indeed see that the sporadic CUDA failure is still reproduced in this PR.

Since the CUDA failure has a separate root cause, I think this PR should stay focused on fixing the SGL byte-length semantics, while keeping the CUDA test skipped for now.

Then the cuda_copy reachability issue can be fixed separately, and the CUDA test can be re-enabled as part of that fix.

Could you also update the PR description accordingly?

[usberkeley]

Hi @michal-shalev I found that the incorrect data size calculation is not the cause of the “CUDA SGL transport protocol not found” error.
This PR will first fix the data size issue. Would it make sense to re-enable the skipped CUDA test first, and fix the actual root cause in a separate PR?

Root cause analysis

The test uses two UCP workers in the same process. cuda_ipc rejects same-process communication, so it should fall back to cuda_copy.
However, each cuda_copy iface gets a different random ID (cuda_copy_iface.c:341), while connection setup requires the IDs to match (cuda_copy_iface.c:89). As a result, the two workers are incorrectly considered unreachable.
Also, although the CI driver supports CUDA 13.0, UCX is built with CUDA 12.8. CUDA IPC only advertises native SGL support when built with CUDA >= 13.0.

Thanks for digging into this. I can indeed see that the sporadic CUDA failure is still reproduced in this PR.

Since the CUDA failure has a separate root cause, I think this PR should stay focused on fixing the SGL byte-length semantics, while keeping the CUDA test skipped for now.

Then the cuda_copy reachability issue can be fixed separately, and the CUDA test can be re-enabled as part of that fix.

Could you also update the PR description accordingly?

Got it, no problem -- Done ❤️

[michal-shalev]

@usberkeley can you please merge master?

[usberkeley]

@usberkeley can you please merge master?

Got it --Done

[michal-shalev]

@usberkeley can you please merge master?

Got it --Done

"This branch is out-of-date with the base branch"
Please merge upstream master.

[usberkeley]

@usberkeley can you please merge master?

Got it --Done

"This branch is out-of-date with the base branch" Please merge upstream master.

Got it -- I’ll keep that in mind next time

[michal-shalev]

LGTM. Thanks for contributing to UCX!

@brminich, can we merge this, or do you have any other comments on your end?

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

this log removal looks unrelated to the SGL length change — why drop it? this branch aborts the request and fails the EP, so the error message is useful for diagnosing the no-protocol case. if it's intentional, pls move it to a separate PR.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

why is the cannot find remote protocol for error removed here? it looks unrelated to the SGL length fix, and it drops the diagnostic that reported strb. can this go in a separate PR (or be kept)?

[svc-ucx]

🤖 CI Triage Agent — UCX PR (Tests roce on worker 0) · commit d38af205

TL;DR: dcx/test_ucp_sockaddr.fail_and_force_close_during_rndv/2 aborted on ucs_assertv((length + min_frag_diff) <= req->send.state.dt_iter.length) in ucp_proto_common_zcopy_adjust_min_frag() — an 8-byte rndv GET fetch request was re-selected (on the wireup pending-replay/restart path) onto rndv/get/zcopy whose min_frag is 65 (mlx5 cap.get.min_zcopy = max_inl_cqe+1), because ucp_proto_request_init() now uses the proto-v1 field req->send.length as the selection length instead of req->send.state.dt_iter.length, which is the only length set for rendezvous reply requests.

Full analysis

Summary: gtest crash (core dumped, make: *** [Makefile:4713: test] Aborted) in dcx/test_ucp_sockaddr.fail_and_force_close_during_rndv/2 <dc_x/tag,sa_data_v1> on the "roce on worker 0" job: proto_common.inl:189 Assertion '(length + min_frag_diff) <= req->send.state.dt_iter.length' failed: req=0x[REDACTED:Hex High Entropy String] length=8 min_frag_diff=57 dt_iter.length=8.

Root cause: The backtrace shows the abort path is protocol re-selection: ucp_wireup_eps_progress → ucp_wireup_replay_pending_request (wireup.c:1224) → ucp_proto_request_restart (proto_common.c:893) → ucp_request_send → ucp_proto_rndv_get_zcopy_fetch_progress (rndv_get.c:188) → ucp_proto_multi_zcopy_progress → ucp_proto_common_zcopy_adjust_min_frag.

• The request's real payload is 8 bytes (dt_iter.length=8, the sizeof(uint64_t) handshake send in listen_and_communicate() → send_recv(), test_ucp_sockaddr.cc:581), yet the selected protocol's mpriv->min_frag is 65 (min_frag_diff = 65 - 8 = 57). min_frag for rndv/get/zcopy comes from cap.get.min_zcopy (rndv_get.c:39-40 → proto_common.c:411 → proto_multi.c:719), which for RC/DC mlx5 is max_inl_cqe+1 = 65.
• A protocol whose min_frag is 65 must never be selected for an 8-byte message — ucp_proto_thresholds_next_range() (proto_select.c:81-94) filters by range->start, i.e. by perf->min_length, which is >= min_frag. So the selection was made with a different length than the request actually carries.
• That is exactly what ucp_proto_request_init() does on the restart path: src/ucp/proto/proto_common.c:858 uses msg_length = req->send.length;. send.length is a proto-v1 field (see ucp_request.h:175 and the assertion "req->send.state.dt should not be used with protov2", ucp_request.inl:584) and is never assigned for rendezvous reply requests — ucp_proto_rndv_receive_start() (proto_rndv.c:907-936) only initializes req->send.state.dt_iter via ucp_datatype_iter_move() and passes rts->size explicitly to ucp_proto_rndv_send_reply(). So on restart the lookup uses a stale/garbage length (the ep also had a 4 KB rndv in flight in this test), lands in the ≥65-byte range, and rndv/get/zcopy with min_frag=65 is chosen for an 8-byte fetch → the min-frag overlap adjustment would read past the request → assert.

Implicated commit: ed5f044b "UCP/RMA: Fix PUT-SGL protocol selection length" (Mason Pitt), with follow-up 4994ff1c "UCP/RMA: Address review comments" — the only recent changes to src/ucp/proto/proto_common.c on the get-sgl branch (head [REDACTED:Hex High Entropy String], merge of upstream master).

File: src/ucp/proto/proto_common.c:858 (ucp_proto_request_init()); assert site src/ucp/proto/proto_common.inl:189; consumer src/ucp/rndv/rndv_get.c:173.

Suggested fix: Don't use the proto-v1 req->send.length as the protocol-selection length for proto-v2 requests. Restore msg_length = req->send.state.dt_iter.length; and handle the SGL case explicitly, e.g.:

    msg_length = (req->send.state.dt_iter.dt_class == UCP_DATATYPE_SGL) ?
                 ucp_datatype_iter_sgl_total_length(&req->send.state.dt_iter) :
                 req->send.state.dt_iter.length;

(or set req->send.length for all proto-v2 requests, including the rndv reply request created in ucp_proto_rndv_receive_start(), so the two lengths can never diverge). Add a regression guard right after selection, e.g. ucs_assertv(msg_length == req->send.state.dt_iter.length || dt_class == UCP_DATATYPE_SGL, ...), so a length mismatch fails loudly at selection time instead of deep inside ucp_proto_common_zcopy_adjust_min_frag(). The test is deterministic enough to reproduce with --gtest_filter=dcx/test_ucp_sockaddr.fail_and_force_close_during_rndv/2.

Related: PR #11778 (get-sgl); prior sibling fixes on the same code path: #11686 "UCP/PROTO/MULTI: Preserve combined minimum for rendezvous", #11690 "Rename single-lane minimum flag", #11765 "Generalize lane SGL zcopy count for put and get".

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id e4e1b097-85c2-4b66-ac9b-a52fbe18e70e in the triage console for the audit trail.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

src/ucp/proto/proto_reconfig.c

This removal looks unrelated to the SGL length change, and it leaves strb built but unused while dropping the failure diagnostic. Can we keep the ucs_error(... strb ...), or drop the strb population as well? Per REVIEW.md, feature/bug-fix PRs should not carry unrelated changes.

[usberkeley]

Hi @michal-shalev
This PR has quite a few conflicts with master, and I’ve force-pushed it a few times. Would it be okay if I just open a fresh PR?

I should have it ready in the next couple of days. I’ll clean things up and double-check everything before submitting.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 0) · commit 41c82102

TL;DR: The ASAN/RoCE gtest shard didn't time out because it was slow — it hung: the multi-fragment AM eager-zcopy send over dc_x (dcx/test_ucp_am_nbx_reply_always.multi_zcopy/0) never completed (no UCT completion, receiver got 0 of 1 messages), so the 875 s request deadline and then the 900 s gtest watchdog fired and aborted the shard with SIGABRT. Investigate the DC AM-zcopy send path exercised by this branch's changes to the shared UCP zcopy/proto code — do not raise any timeout.

Full analysis

Summary: make -C test/gtest test aborted (core dumped) because dcx/test_ucp_am_nbx_reply_always.multi_zcopy/0 <dc_x> hung for the whole remaining job; the gtest watchdog killed the process.

Root cause: Evidence from the job log:

• 16:09:28 [ RUN ] dcx/test_ucp_am_nbx_reply_always.multi_zcopy/0 <dc_x> — last meaningful line.
• Then a 14.6-minute gap with zero output (the single largest gap in the log, i.e. a hang, not slow progress), ending at 16:24:06 with ucp_test.cc:300 UCX ERROR request 0x62e00000a420 did not complete on time and test_ucp_am.cc:542: Expected equality ... m_recv_counter Which is: 0 / m_send_counter Which is: 1.
• A second request also stalls (16:35:22, request 0x62e00000a2e0 did not complete on time), then at 16:39:06 the gtest watchdog (900 s, test_helpers.cc:57 Connection timed out - abort testing) sends SIGABRT → Caught signal 6 → make: *** [Makefile:4713: test] Aborted (core dumped).

So a single UCP request in the AM eager zcopy multi-fragment path over dc_x never completed and nothing was ever delivered (recv counter 0). The test (test/gtest/ucp/test_ucp_am.cc:1124, ZCOPY_THRESH=1, RNDV_THRESH=inf, REPLY flag, size = am.max_zcopy - hdr - reply_ftr + 1) forces exactly two zcopy fragments, so either the first uct_ep_am_zcopy completion was lost / the request sat forever on a DC pending queue, or the second fragment was never posted. This is the first dc_x data transfer in the shard (the earlier dcx/* cases were [SKIP] (Address sanitizer)), i.e. a deterministic-looking stall on the DC zcopy path, not a mid-run flake.

Implicated commit: Not provable from the log alone. Candidates, in order of suspicion:

• This PR's own changes to code shared by all zcopy/multi protocols (branch get-sgl, head 41c82102): ed5f044b and 4994ff1c (Mason Pitt, src/ucp/proto/proto_common.c, src/ucp/proto/proto_common.inl) and 1df16ede (Mason Pitt, src/ucp/dt/datatype_iter.inl). Note the new hard-error path ucp_proto_common_get_lane_perf() → return UCS_ERR_INVALID_PARAM when params->min_length > tl_max_frag, which propagates out of ucp_proto_multi_query_lanes()/ucp_proto_multi_init() for every multi-lane protocol, and the zcopy cleanup mask change in ucp_proto_request_zcopy_complete_cb().
• Master commits pulled in by merge 41c82102 (2026-09-01) that touch the DC/RC mlx5 send-completion path: [REDACTED:Hex High Entropy String] "UCT/IB/MLX5: support in progress error handler (UCT/IB/MLX5: support in progress error handler #11829)", 8282944a "UCT/IB/MLX5: Track RC send PSN with path MTU (UCT/IB/MLX5: Track RC send PSN with path MTU #11821)", c8495100 "UCT/IB: Add configurable TTL knob for the AH cache (UCT/IB: Add configurable TTL knob for the AH cache #11847)" (DC AH lifetime).

File: test/gtest/ucp/test_ucp_am.cc:1124 (hanging test) / test/gtest/ucp/test_ucp_am.cc:542 (reported assert); implicated product code: src/ucp/proto/proto_multi.inl:337 (ucp_proto_am_zcopy_multi_common_send_func), src/ucp/proto/proto_common.c:372-378, src/ucp/proto/proto_common.inl:116-129.

Suggested fix:

1. Reproduce locally/on the same node with only that test to confirm determinism:
   UCX_TLS=dc_x ./gtest --gtest_filter='dcx/test_ucp_am_nbx_reply_always.multi_zcopy*' under ASAN, with UCX_PROTO_INFO=y UCX_LOG_LEVEL=trace_req to see which protocol was selected and whether the second fragment is posted or the request is parked in a DC pending queue (ucp_proto_multi_no_resource).
2. Bisect the two suspicion sets: run the same filter on plain upstream master (the merge base) — if it hangs there too, the regression is one of [REDACTED:Hex High Entropy String] / 8282944a / c8495100 and this PR is only the messenger; if it passes, revert/re-check the PR's edits in proto_common.c (the new UCS_ERR_INVALID_PARAM return, which turns a "lane unsuitable" condition into a hard init error for all multi protocols) and the datatype_iter.inl changes, and make the length/iov checks affect only the SGL protocols they were intended for.
3. Do not increase the gtest/job time limit — the log shows a stalled request, not work in progress.

Related: PR #11778 "UCP/RMA: Fix PUT-SGL protocol selection length"; earlier SGL work #11765, #11677, #11647; possibly relevant recent DC/mlx5 PRs #11829, #11821, #11847.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id e0cbb808-2bdd-46af-bf52-fd9e2bee3980 in the triage console for the audit trail.