[tvegas1]

What?

Add support for copy-out to non-host memory.

Why?

When reading to CUDA memory, get/bcopy cannot be used.

How?

Add copy-out and use temporary memory descriptor area, to handle fragments potentially reordered.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

this adds malloc/free per GET request plus O(length/max_frag) memory (~200KB for a 100MB get on an 8KB frag lane), and get/bcopy is selected for small messages too, where docs/OptimizationStyle.md requires no malloc/free on the data path. can we keep the host path allocation-free (pass dest with the plain memcpy unpack cb) and use a pooled ctx only when the local memory is not CPU-accessible?

[svc-nvidia-pr-review]

can reset/abort run while fragments posted earlier are still in flight (pending replay from ucp_wireup_replay_pending_requests(), or a send error on a later fragment)? in that case UCT is left with a dangling unpack arg - IMO the array should be released only in the completion callback. (also applies to the free at :201)

[svc-nvidia-pr-review]

why keep the unpack ctx array in req->send.buffer? it is the user send buffer field and get_am sets it for get_reply processing - pls use a dedicated field or document the aliasing.

[svc-nvidia-pr-review]

max_frag is always > 0 (asserted in ucp_proto_multi_init_priv()), pls drop this check and the one in the progress function (:119); this one also adds a branch on the send path.

[svc-nvidia-pr-review]

minor: num_frags is always >= 1 (zero-length get returns early in ucp_get_nbx()), and if it were 0 the frag_idx < num_frags assert in the send function would fail anyway - pls use num_frags directly instead of ucs_max(num_frags, 1).

[svc-ucx]

🤖 CI Triage Agent — UCX PR (Tests roce on worker 1) · commit e1756bc3

TL;DR: The roce on worker 1 job did not fail a test — ucx_perftest hung for ~16 minutes in the very first batch test (ucp_contig_tag_lat/2048, UCX_TLS=rc_verbs,rc_x on mlx5_0:1) and the job was then killed by an agent shutdown; the perftest/mpirun processes were still alive at cleanup, so this is a UCP connection-establishment/first-message hang, not a slow test.

Full analysis

Summary: mpirun -np 2 ucx_perftest -b test_types_short_ucp … produced the results table header and the +ucp_contig_tag_lat/2048 stage line at 11:54:23, then emitted nothing for 15m56s until Azure cancelled the job (agent shutdown) at 12:10:19 and terminated the still-running mpirun + two ucx_perftest PIDs.

Root cause: A hang inside UCX during the first UCP tag ping-pong over RC/RoCE. Evidence from the log:

• Last application output: parser.c:2578 UCX WARN unused environment variable: UCX_TCP_PORT_RANGE from both ranks (892315, 892319) at 11:54:23.165 — i.e. both workers had just been created and the first test was starting.
• Largest inter-line gap in the whole step = 15m56s (11:54:23 → 12:10:19), with zero output. A 2048-byte tag_lat with -n 100 -w 1 normally completes in well under a second, and the batch would have printed dozens more stage lines.
• Cleanup shows Terminate orphan process: pid (892263) (mpirun), (892315) (ucx_perftest), (892319) (ucx_perftest) — the processes were alive and stuck, not slow.
• The ##[error]The agent has received a shutdown signal / The operation was canceled lines are the symptom (host/agent taken down while the job was wedged), not the cause. The second failed job in this build (roce on worker 3) was cancelled by the same event at the same timestamp.

The hung operation is an eager tag send/recv (2048 B is below the RC rendezvous threshold), so the code this PR actually changes — the offloaded get/bcopy protocol in src/ucp/rma/get_offload.c (memtype-aware unpack via ucp_dt_contig_unpack, per-fragment unpack-context array stored in req->send.buffer) — is not reachable in the hung test. The stall is therefore most likely in the wireup/RC path that the branch picked up when master was merged in ([REDACTED:Hex High Entropy String], "Merge branch 'master' into get_bcopy_to_non_host", 11:46 the same morning). The prime suspect on that path is fb88400f — "UCP/WIREUP: Add a token trailer and an ACK message to the lane address exchange (#11843)", which makes EP readiness depend on a new wireup ACK; a lost/never-sent ACK produces exactly this signature (both peers alive, first tag operation never completes). This is a hypothesis about which component to look at — the log contains no stack trace, so it cannot be pinned from CI output alone.

Implicated commit: unknown from the log; candidates are fb88400f (Evgeny Leksikov, UCP/WIREUP ACK, #11843) and 858ffeae (Thomas Vegas, wireup p2p lane matching, #11912), both pulled into the branch by merge [REDACTED:Hex High Entropy String] (Mikhail Brinskiy). The PR's own change (src/ucp/rma/get_offload.c) is not on the hung code path.

File: src/ucp/wireup/wireup.c (wireup ACK/lane-address exchange) — hang site; CI-side gap: contrib/test_jenkins.sh:643 ($MPIRUN … $ucx_perftest $ucp_test_args runs with no per-invocation timeout, so a hang consumes the whole job with no diagnostics).

Suggested fix:

1. Do not raise the job/step time limit — the process was hung, not slow.
2. Reproduce on the same host with the exact command (UCX_NET_DEVICES=mlx5_0:1 UCX_TLS=rc_verbs,rc_x ucx_perftest -t tag_lat -s 2048 -n 100 -w 1) and capture UCX_LOG_LEVEL=debug, UCX_PROTO_INFO=y plus gstack/gdb -p backtraces of both ucx_perftest PIDs to identify whether both peers are spinning in ucp_worker_progress waiting for wireup completion (would confirm the wireup-ACK suspicion) or in RC completion polling (fabric/QP issue on the RoCE port).
3. Re-run the roce stage on plain master at the merge base; if master also hangs, this PR is a bystander and UCP/WIREUP: Add a token trailer and an ACK message to the lane address exchange #11843 should be revisited (ensure the wireup ACK is retransmitted/flushed and that data can flow when the ACK path is not yet established).
4. CI hardening: wrap the perftest invocations in timeout <N>m and, on timeout, dump stacks of the surviving processes, so future hangs report the hang location instead of appearing as an agent cancellation.

Related: PR #11416 (this branch, "UCP/RMA/BCOPY: Support non-host memory for get/bcopy"); #11843 (wireup token trailer + ACK, prime suspect); #11912 (wireup p2p lane matching diagnostics); #11840 (get/bcopy failover, same protocol as this PR).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 84621168-119b-4881-bdb9-6bf74894a77a in the triage console for the audit trail.