UCT/CUDA/COPY: Qualify cuda_copy for RNDV with peer-failure EH by pentschev · Pull Request #11319 · openucx/ucx

pentschev · 2026-04-01T15:05:30Z

What?

Allow endpoints connected to the same UCP worker to select transports such as cuda_copy when using UCP_ERR_HANDLING_MODE_PEER, even if those transports do not support peer-failure handling or RMA invalidation.

For same-worker PEER endpoints, wireup relaxes the peer-failure requirements during lane selection while retaining the requested PEER mode in the endpoint configuration. When a request buffer must be invalidated, UCP skips only RMA-BW lanes whose memory domain lacks UCT_MD_FLAG_INVALIDATE_RMA, while capable lanes remain in the invalidation map.

Add a same-worker peer-error wireup test. Keep the existing RMA-invalidation test connected to a different worker so it continues to check the cross-worker requirement.

UCP_ERR_HANDLING_MODE_FAILOVER and cross-worker endpoints retain their existing requirements.

Why?

With peer error handling and rendezvous enabled, lane selection normally requires peer-failure handling and RMA invalidation. cuda_copy does not provide those capabilities, so it was excluded even when both endpoint addresses refer to the same worker.

Relaxing selection for that case permits the local copy path. Preserving invalidation for capable lanes also protects against in-flight RMA on network lanes that may be selected by a same-worker endpoint.

Closes #11318

pentschev · 2026-04-09T12:53:10Z

Thanks @yosefe @shasson5 . I've reverted the UCT changes and pushed a fix on the UCP-layer. Could you please review?

yosefe

test failure seems relevant

yosefe · 2026-04-10T12:42:58Z

    if (ep_init_flags & UCP_EP_INIT_ERR_MODE_PEER_FAILURE) {
+        /* No independent remote worker when connecting an EP to itself (loopback,
+         * memtype EPs, etc.): peer-failure iface caps and MD invalidation are
+         * relaxed in wireup; see ucp_request_get_invalidation_map(). */
+        if ((unpacked_addr != NULL) && (unpacked_addr->uuid == worker->uuid)) {
+            return;
+        }


IMO it would be better to update ucp_ep_err_mode_init_flags to not set UCP_EP_INIT_ERR_MODE_PEER_FAILURE for uuid==remote_uuid case

Updated. For UCP_ERR_HANDLING_MODE_PEER, ucp_ep_err_mode_init_flags() now omits UCP_EP_INIT_ERR_MODE_PEER_FAILURE when the remote UUID matches the local worker UUID. The effective flags are applied only during lane selection, so the endpoint configuration still retains UCP_ERR_HANDLING_MODE_PEER.

I intentionally kept the full requirements for FAILOVER: removing peer-failure handling there makes the PSN failover protocol unavailable for same-UUID endpoints. Added same-process wireup and optional-RMA-invalidation coverage as well.

svc-ucx · 2026-08-24T17:27:40Z

🤖 CI Triage Agent — UCX PR · commit e2aa0130

TL;DR: Both codestyle jobs ("AUTHORS file update check" and "commit title") failed because the PR commit range SourceVersion^1..SourceVersion^2 resolved to ~100 unrelated upstream master commits (24 different authors) instead of just this PR's commits — a consequence of the shallow fetchDepth: 100 checkout plus a branch whose merge-base is deeper than that window. Rebase intra-process-cuda-copy-fix onto current master (and/or fix the CI range computation to use git merge-base with a deep-enough fetch).

Full analysis

Summary: AUTHORS file check step exited 1 (git diff --exit-code after contrib/authors_update.sh) because the script appended shasson5 <103439971+shasson5@users.noreply.github.com> and wehy <18174858434@163.com> to AUTHORS; the parallel commit title job failed on the same commit range.

Root cause: The step computes base_revision=$(git rev-parse [REDACTED:Hex High Entropy String]^1) (= master tip [REDACTED:Hex High Entropy String]) and head_revision=cf6095ed^2 (= PR head e2aa01302), then runs git log --first-parent --no-merges base..head. The log shows this range yielding 24 authors — Yossi Itigin, Thomas Vegas, Artemy Kovalyov, Alexey Rivkin, Raul Akhmetshin, etc. — which are clearly upstream master commits, not commits from a single-purpose "intra-process cuda copy fix" PR. The checkout is --depth=100 from the merge commit (git fetch ... --depth=100 +cf6095ed... in the log), so git cannot see the true merge-base of bade18369 and e2aa01302; ancestry exclusion is cut off at the shallow graft and master's history leaks into the range. Two of those leaked authors are not matched by authors_update.sh's grep (shasson5's GitHub noreply address doesn't match the existing Shachar Hasson <shasson@nvidia.com> entry; wehy is absent), so the script rewrote AUTHORS and git diff --exit-code failed. The commit title job uses the identical range (codestyle_check_commit_title "$base_revision..$head_revision"), so it flags titles of those same unrelated upstream/merge commits.

Implicated commit: cfd9cd1f — "BUILD/CI: Limit PR commit range checks (#11527)", Yossi Itigin, 2026-08-17 (three days before this build; it is the only recent change to both buildlib/pr/codestyle.yml and contrib/authors_update.sh).

File: buildlib/pr/codestyle.yml:88-92 (AUTHORS job) and buildlib/pr/codestyle.yml:21-24 (commit_title job); helper at contrib/authors_update.sh:18 and buildlib/tools/codestyle.sh:11.

Suggested fix:

Immediate unblock for PR UCT/CUDA/COPY: Qualify cuda_copy for RNDV with peer-failure EH #11319: rebase the branch onto current openucx/master and force-push, so the merge-base falls inside the 100-commit fetch window. Re-run the pipeline.
CI hardening in buildlib/pr/codestyle.yml — make the range robust to shallow clones, e.g.:
```
git fetch --no-tags --deepen=1000 origin || git fetch --unshallow origin || true
head_revision=$(git rev-parse "$(Build.SourceVersion)^2")
base_revision=$(git merge-base "$(Build.SourceVersion)^1" "$head_revision")
```
and use "$base_revision..$head_revision" for both authors_update.sh and codestyle_check_commit_title. Optionally bail out with a clear "branch too far behind master, please rebase" message when git merge-base fails, instead of silently checking 100 upstream commits.
Separate cleanup: add a .mailmap entry mapping 103439971+shasson5@users.noreply.github.com → Shachar Hasson <shasson@nvidia.com> (and have authors_update.sh honor .mailmap via %aN%x09%aE) so GitHub noreply addresses don't create duplicate AUTHORS entries.

Related: #11527 (introduced the current range computation), #11319 (this PR)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id efdfa07a-6d8e-45f0-881d-f335da9807d5 in the triage console for the audit trail.

shasson5 · 2026-08-24T17:34:08Z

ok from my side.

gleon99 · 2026-08-25T09:09:43Z

@rakhmets please review

Advertise UCT_IFACE_FLAG_ERRHANDLE_PEER_FAILURE so ucp_wireup_fill_peer_err_criteria does not exclude cuda_copy from RMA BW lanes. Advertise UCT_MD_FLAG_INVALIDATE and UCT_MD_FLAG_INVALIDATE_RMA and handle UCT_MD_MEM_DEREG_FLAG_INVALIDATE in mem dereg (invoke completion), matching ucp_wireup_add_rma_bw_lanes and ucp_memh_dereg expectations.

When an endpoint is wired to the same UCP worker (same unpacked address UUID), there is no independent remote peer for cross-worker RMA. Skip requiring UCT_IFACE_FLAG_ERRHANDLE_PEER_FAILURE and UCT_MD_FLAG_INVALIDATE_RMA in RMA BW wireup criteria in that case. In ucp_request_get_invalidation_map(), return an empty invalidation map for UCP_EP_CONFIG_KEY_FLAG_SELF so RMA BW lanes without MD invalidate support (e.g. cuda_copy) remain valid for same-worker / memtype EPs with peer-failure error handling. Extend ucp_wireup_fill_peer_err_criteria() and ucp_wireup_fill_aux_criteria() with worker + unpacked address so auxiliary and other lane selection paths apply the same rule.

Derive effective wireup flags once from the requested error mode and\nthe remote worker UUID. Same-worker endpoints no longer require peer\nfailure or RMA invalidation transport capabilities, while their endpoint\nconfiguration retains the requested error mode.\n\nAdd self-transport coverage for peer error handling.

…a-copy-fix

pentschev · 2026-08-25T10:33:39Z

I have now only rebased to resolve the codestyle errors, there were no code changes.

pentschev · 2026-09-02T07:39:37Z

@rakhmets anything to do here?

svc-nvidia-pr-review · 2026-09-02T07:40:08Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-09-02T07:42:41Z

Question on self + FAILOVER: ucp_ep_err_mode_init_flags(UCP_ERR_HANDLING_MODE_FAILOVER, 1) still returns the full UCP_EP_INIT_ERR_MODE_FAILOVER_MASK (confirmed by the new init_flags_self test), so a same-worker FAILOVER endpoint still imposes the peer-failure transport requirement that this PR relaxes for PEER mode. Is that intentional (FAILOVER on loopback being unsupported/not a target), or should self also relax FAILOVER? A one-line comment on why only PEER is relaxed would help.

svc-nvidia-pr-review · 2026-09-16T15:08:17Z

Residual coverage gap (noted in an earlier round): there is still no test that runs traffic and triggers error handling on a self PEER endpoint, so the ucp_request_get_invalidation_map() short-circuit is not exercised end to end.

svc-nvidia-pr-review · 2026-09-16T15:58:33Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-09-16T16:08:18Z

    key->dst_md_cmpts = dst_md_storage;

-    status = ucp_wireup_select_lanes(ep, ep_init_flags, *tl_bitmap,
+    select_ep_init_flags = ep_init_flags;


this repeats the address->uuid == worker->uuid check that ucp_wireup_select_lanes() already does to set UCP_EP_CONFIG_KEY_FLAG_SELF, and it only covers the worker-address path; wireup_cm.c:212 calls ucp_wireup_select_lanes() directly and keeps the peer-failure requirement, while the ucp_request.c change already treats such EPs as "no invalidation needed". Can we relax the flag inside ucp_wireup_select_lanes() so the CM path behaves the same?

Not moving the relaxation into ucp_wireup_select_lanes(). Its CM caller constructs an intermediate client configuration from a locally packed dummy address before the actual peer address is known, that address has the local UUID by construction. Applying the relaxation there would incorrectly weaken peer-failure requirements for cross-worker CM EPs. The current check is intentionally limited to worker-address selection, where the UUID is the actual peer’s UUID.

svc-nvidia-pr-review · 2026-09-16T17:21:05Z

🤖 Starting review — findings will be posted here when done.

svc-ucx · 2026-09-17T01:38:29Z

🤖 CI Triage Agent — UCX PR (AddressSanitizer gpu on worker 3) · commit 5f041ecf

TL;DR: The ASAN‑GPU gtest run failed on a single test, dcx/test_ucp_am_nbx_seg_size.multi/3 <dc_x/proto_v1/reply>, which stalled ~200 s (all 10 185 other tests passed in ms) — the receiver never finished assembling the multi‑fragment eager AM, so wait_receives() hit the 180 s test timeout; this path (host memory, RNDV_THRESH=inf, no peer‑failure EH) is not touched by PR #11319's cuda_copy change, so it is a hang/flake in multi‑fragment AM assembly rather than a PR regression — re‑run the job and, if it reproduces, debug the receiver‑side assembly in ucp_am_handle_unfinished().

Full analysis

Summary: make test in the "AddressSanitizer gpu on worker 3" job exited 2 because exactly one gtest failed: dcx/test_ucp_am_nbx_seg_size.multi/3, where GetParam() = dc_x/proto_v1/reply.

Root cause: Evidence from the log: [ FAILED ] 1 test out of 10 186, and the TOP‑20 list shows dcx/test_ucp_am_nbx_seg_size.multi/3 - 200481 ms, while every neighbouring test in the same log completes in 0–300 ms and the whole suite averages ms per test. That 200 s is a stall, not slow progress: the test sends one AM of seg_size()*2 bytes (multi‑fragment eager, RNDV_THRESH=inf, PROTO_ENABLE=n, UCP_AM_SEND_FLAG_REPLY) and then blocks in wait_receives() → wait_for_value(&m_recv_counter, m_send_counter), which gives up after ucs::test_timeout_in_sec = 180 (test/gtest/common/test_helpers.cc:29) and then trips EXPECT_EQ(m_recv_counter, m_send_counter). So one or more fragments of the long AM were never delivered/assembled on the receiver — a receiver-side assembly stall on the dc_x path. Note that this test also intentionally creates a third entity with a randomized IB_SEG_SIZE (1 KB–64 KB) that differs from the receiver's, relying on ADDRESS_VERSION=v2 seg-size negotiation, and that ucp_am_handle_unfinished() only checks the head of ep_ext->am.started_ams and returns early if it is incomplete — a single stuck/partial first_rdesc there blocks delivery of everything behind it. Nothing in this path involves CUDA memory, rendezvous, or peer-failure error handling, i.e. the code changed by PR #11319 (re-adding UCT_IFACE_FLAG_ERRHANDLE_PEER_FAILURE to cuda_copy — currently absent at src/uct/cuda/cuda_copy/cuda_copy_iface.c:107-112 after the earlier revert b020c4f) is not exercised by this test, so the failure is almost certainly pre-existing flakiness and not caused by the PR.

Implicated commit: unknown (not PR #11319's cuda_copy change; suspect area is the interval-tree/PSN multi-fragment AM assembly introduced/modified by ef265cd "Multi Eager PSN Protocol" and 2bafbb0 "AM/FT: resend flag for first fragment retransmissions", both by Evgeny Leksikov / shasson5)

File: test/gtest/ucp/test_ucp_am.cc:1710 (test_ucp_am_nbx_seg_size.multi) → test/gtest/ucp/test_ucp_am.cc:534 (wait_receives()); receiver-side assembly: src/ucp/core/ucp_am.c:1457-1472 (ucp_am_handle_unfinished, head-of-list early return)

Suggested fix: Re-trigger the ASAN GPU job to confirm the failure is unrelated to this PR (expect it to pass; the PR touches only cuda_copy iface caps, and this test uses host memory with RNDV_THRESH=inf). In parallel, file/track the hang separately: reproduce with --gtest_filter=dcx/test_ucp_am_nbx_seg_size.multi/3 plus UCX_LOG_LEVEL=debug and a fixed IB_SEG_SIZE sweep to see whether a fragment is dropped (look for "dropped duplicate message ..." from ucp_am_is_duplicate_psn, src/ucp/core/ucp_am.c:1758) or whether an incomplete first_rdesc sits at the head of started_ams. A robustness fix worth considering is making ucp_am_handle_unfinished() continue scanning the list instead of returning on the first incomplete entry, so one stalled partial message cannot block delivery of already-assembled ones. Do not raise the gtest timeout — the log shows a stall, not legitimate slowness.

Related: PR #11319 (the PR under test; earlier attempt [REDACTED:Hex High Entropy String] was reverted by b020c4f); no existing issue found for test_ucp_am_nbx_seg_size hangs.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 8794778a-050a-42ea-bdca-7c5450d18805 in the triage console for the audit trail.

svc-nvidia-pr-review · 2026-09-28T08:27:52Z

🤖 Starting review — findings will be posted here when done.

svc-ucx · 2026-09-28T10:37:14Z

🤖 CI Triage Agent — UCX PR (Tests gpu on worker 0) · commit 00e37ec6

TL;DR: The gtest job failed on exactly one test — shm_ib/test_ucp_proto_am_rndv.am_explicit_scheme_disables_force_provenance/0 — a protocol‑selection assertion that hard‑fails (rather than skips) when am/rndv cannot be selected for CUDA memory under RNDV_SCHEME=get_zcopy on this GPU worker; it is unrelated to PR #11319's self‑lane invalidation change and should be guarded/skipped like its tag/RMA counterparts.

Full analysis

Summary: make test in contrib/test_jenkins.sh exited non‑zero because 1 of 10306 gtests failed: shm_ib/test_ucp_proto_am_rndv.am_explicit_scheme_disables_force_provenance/0 (GetParam() = shm,ib,cuda_copy,rocm_copy); 10285 passed, 20 skipped, no crash, timeout or sanitizer error.

Root cause: The test runs check_am_rndv_remote_proto_config(0, 0) with RNDV_PIPELINE_SHM_CUDA_STAGING_FORCE=y and RNDV_SCHEME=get_zcopy. Inside select_am_rndv_remote_proto_config() the test hard-asserts that the AM send protocol picked at 1 MB for CUDA memory is exactly "am/rndv" (EXPECT_NE(nullptr, select_elem), EXPECT_STREQ("am/rndv", ...)), and then ASSERT_NE(nullptr, remote_proto_config). Because RNDV_SCHEME=get_zcopy sets context->config.ext.rndv_mode != UCP_RNDV_MODE_AUTO, ucp_proto_rndv_shm_pipeline_force_enabled() (src/ucp/rndv/proto_rndv.inl:20‑28) disables the forced pipeline and the remote RNDV‑recv protocol must be selectable via get_zcopy for a CUDA buffer; if no network MD on this worker can register/reach the CUDA buffer, ucp_proto_rndv_ctrl_select_remote_proto() returns UCS_ERR_UNSUPPORTED (src/ucp/rndv/proto_rndv.c:271‑276), am/rndv is not chosen, and the assertions fire. The test only calls require_cuda_rndv_support() — unlike the analogous tag tests which call require_cuda_net_md() — and unlike the RMA variant, which UCS_TEST_SKIPs when no protocol is found. So this is an environment-sensitive test-robustness gap, not a product regression from this PR: the PR's commits (00e37ec6 "UCP/CORE: Preserve invalidation on capable self lanes", plus the other self/wireup commits on the branch) only touch self-lane wireup/invalidation, which is orthogonal to AM rendezvous protocol selection over shm,ib. Corroborating evidence: three closed TEMP PRs (#11858, #11861, #11864) exist purely to diagnose this same test family's Azure protocol-selection failures introduced with #11685. Note the exact gtest assertion text sits earlier in the 5213 s log than the retrievable tail, so the specific failing EXPECT_* line is inferred from the code path rather than quoted.

Implicated commit: 0311ed79 — jeffnvidia, "UCP/PROTO/RNDV: Isolate remote system device namespace (#11685)" (test family originally added by d9c844ae / 9cc08081, Nathan Bellalou, #11641). Not 00e37ec6 (PR #11319).

File: test/gtest/ucp/test_ucp_proto.cc:805 (test body) → test/gtest/ucp/test_ucp_proto.cc:426‑441 and :448‑459 (failing assertions)

Suggested fix: Make the AM variant as tolerant as the RMA/tag variants: (1) add require_cuda_net_md() to the test_ucp_proto_am_rndv am_* tests (or at least to am_explicit_scheme_disables_force_provenance), and (2) in select_am_rndv_remote_proto_config() replace the hard EXPECT_NE(nullptr, select_elem) / EXPECT_STREQ("am/rndv", ...) with UCS_TEST_SKIP_R("AM/RNDV protocol is not supported") when the selected protocol is not am/rndv, mirroring select_rma_rndv_remote_proto_config() (test_ucp_proto.cc:352‑354). Since the failure is independent of PR #11319, re-run the GPU job to confirm, and file/attach the fix to the #11685 follow-up rather than blocking this PR.

Related: #11685, #11858, #11861, #11864, #11641

gleon99 · 2026-09-28T16:14:21Z

@pentschev please address and fix

svc-nvidia-pr-review · 2026-09-28T21:06:48Z

🤖 Starting review — findings will be posted here when done.

svc-ucx · 2026-09-28T21:16:07Z

🤖 CI Triage Agent — UCX PR (Cuda Create UCX source tarball) · commit 13b658e0

TL;DR: The "Create UCX source tarball" job failed in PublishBuildArtifacts because the GPUNetIO submodule clone got an "Empty reply from server" from github.com, leaving src/uct/ib/mlx5/gdaki/gpunetio as a dangling symlink that the artifact uploader cannot read; fix by retrying/hardening the submodule fetch in autogen.sh and excluding the broken symlink (and .git) from the published artifact.

Full analysis

Summary: Publish of the ucx_src_137319 artifact failed with System.IO.FileNotFoundException: Could not find file '.../s/src/uct/ib/mlx5/gdaki/gpunetio' after the GPUNetIO submodule clone failed during ./autogen.sh.

Root cause: Infrastructure/transient network failure, amplified by non-fail-fast submodule handling — not related to the PR's CUDA copy changes:

21:09:32 Cloning into 'external/gpunetio'... fatal: unable to access 'https://github.com/NVIDIA-DOCA/gpunetio/': Empty reply from server → Clone of ... into submodule path 'external/gpunetio' failed.
autogen.sh:28-29 swallows the error (git submodule update --init external/gpunetio || echo "WARNING: ... continuing..."), so the build proceeds with external/gpunetio empty.
.gitmodules maps submodule name src/uct/ib/mlx5/gdaki/gpunetio to path external/gpunetio; the in-tree src/uct/ib/mlx5/gdaki/gpunetio is a symlink into that empty submodule dir, so it becomes a dangling link (it is also unreadable in the repo checkout).
The Azure PublishBuildArtifacts task walks the whole workspace (1637 files, including .git/objects/pack/...) and hard-fails when it tries to open the broken symlink: 1 files failed to upload, retry these files after a minute → ##[error]File upload failed even after retry. Autogen itself succeeded, so the only failure is the artifact upload.

There is no hang: the largest log gap is the ~60 s built-in upload retry sleep, all other lines are continuous.

Implicated commit: d9b582e8 "UCT/GDA: Use gpunetio submodule (#11122)" — Artemy Kovalyov (submodule + symlink layout); eae153e6 "BUILD: Log an error if GPUNetIO cannot be updated (#11378)" — Thomas Vegas (kept the non-fatal warning). PR commit [REDACTED:Hex High Entropy String] is not implicated.

File: autogen.sh:28 (non-fatal submodule update) and .gitmodules:1-3 (symlink target src/uct/ib/mlx5/gdaki/gpunetio → external/gpunetio)

Suggested fix:

Make the submodule fetch resilient instead of silently degraded, e.g. in autogen.sh:

for i in 1 2 3; do
    git submodule update --init external/gpunetio && break
    echo "WARNING: GPUNetIO submodule update failed (attempt $i), retrying..."
    sleep 5
done

Make the tarball step robust to a missing submodule: in the pipeline's publish step, don't publish the raw workspace — create the tarball first (git archive/make dist) or add a Contents/exclude filter that skips .git/** and src/uct/ib/mlx5/gdaki/gpunetio when external/gpunetio is empty (e.g. rm -f src/uct/ib/mlx5/gdaki/gpunetio guard, or find -xtype l -delete before publishing). This also avoids uploading the 50 MB .git pack.
Re-run the job to confirm it was the transient github.com failure.

Related: PR #11319 (this build); submodule-introducing PRs #11122, #11094, and #11378 (GPUNetIO update error logging). No existing issue found tracking the broken-symlink artifact upload failure — worth opening one.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 3c51d294-3dea-4fef-856e-814d1eb8c76f in the triage console for the audit trail.

svc-nvidia-pr-review · 2026-09-28T21:17:58Z

        if (!ucp_ep_is_lane_p2p(ep, lane)) {
+            /* Same-worker PEER selection may include MDs without RMA
+             * invalidation, while capable MDs still need it. */
+            if ((key->flags & UCP_EP_CONFIG_KEY_FLAG_SELF) &&


maybe check only UCT_MD_FLAG_INVALIDATE_RMA here and drop the SELF/err_mode conditions? They duplicate the selection rule from ucp_wireup_try_select_lanes(), so if the relaxation is later extended (e.g. to FAILOVER, as asked in the earlier thread), this assert starts firing again.

I'm keeping the SELF/PEER guard because lane selection relaxes INVALIDATE_RMA only for that case; FAILOVER was explicitly kept under full requirements. Removing it would silently skip invalidation if an unsupported lane appeared on a cross-worker or FAILOVER endpoint. If selection policy changes later, both paths should change together.

svc-nvidia-pr-review · 2026-09-29T06:24:45Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-09-29T06:37:01Z

        lane = key->rma_bw_lanes[i];

        if (!ucp_ep_is_lane_p2p(ep, lane)) {
+            /* Same-worker PEER selection may include MDs without RMA


minor: can we key the skip on the MD capability instead of re-deriving the wireup relaxation here? It keeps the check tied to the precondition of the assert below, and UCP_EP_CONFIG_KEY_FLAG_SELF is also set for the CM intermediate config, which is selected against the local worker address (wireup_cm.c:190), so it is not by itself a reliable "same worker" indicator.

Suggested change

/* Same-worker PEER selection may include MDs without RMA

if (!(ucp_ep_md_attr(ep, lane)->flags &

UCT_MD_FLAG_INVALIDATE_RMA)) {

/* Same-worker PEER selection may pick MDs without RMA

* invalidation */

ucs_assert(key->flags & UCP_EP_CONFIG_KEY_FLAG_SELF);

continue;

}

The preliminary CM key can be marked SELF, but it selects lanes with the original peer-failure flags, so with RNDV enabled it still requires INVALIDATE_RMA. The proposed check would not distinguish that key and would also skip unsupported FAILOVER lanes. I’m keeping the exception limited to SELF+PEER, the case whose selection requirements this PR relaxes.

Reverted the UCT changes and pushed a fix on the UCP-layer.

`waitRequests()` progressed the worker before checking completion. In blocking progress mode, requests that completed inline leave nothing to wake `epoll_wait()`, so each wait lasted a full UCX keepalive interval (20 s). Surfaced with UCX master: since openucx/ucx#11319, same-worker endpoints use `self`/`cma` lanes for RMA, so GET and flush complete inline. Tested against UCX master: all `RequestTest.*` pass, 20 s stalls gone (100 -> 0). Failure log: https://dev.azure.com/ucfconsort/ucx/_build/results?buildId=138700&view=logs&j=d476be9b-0898-5876-257d-321aec8cffde&t=709b8934-27f9-5e64-88db-40e26fd98cd3&l=7506 Authors: - Alexey Rivkin (https://github.com/Alexey-Rivkin) - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) URL: #777

gleon99 requested review from iyastreb and shasson5 April 4, 2026 13:41

yosefe previously requested changes Apr 9, 2026

View reviewed changes

Comment thread src/uct/cuda/cuda_copy/cuda_copy_iface.c Resolved

pentschev force-pushed the intra-process-cuda-copy-fix branch from eec6960 to 4fd4eb4 Compare April 9, 2026 12:54

yosefe reviewed Apr 10, 2026

View reviewed changes

openucx deleted a comment from svc-nixl May 4, 2026

shasson5 reviewed Aug 20, 2026

View reviewed changes

Comment thread src/ucp/wireup/wireup.c Resolved

Comment thread src/ucp/wireup/wireup.c Resolved

shasson5 reviewed Aug 20, 2026

View reviewed changes

Comment thread src/ucp/core/ucp_request.c Resolved

openucx deleted a comment from kussberg Aug 20, 2026

gleon99 requested a review from rakhmets August 25, 2026 09:09

pentschev added 8 commits August 25, 2026 02:59

UCT/CUDA/COPY: Revert qualify cuda_copy for RNDV with peer-failure EH

b020c4f

UCP/EP: Fix self recovery auxiliary transport selection

29e6fb6

UCP/WIREUP: Preserve failover requirements for self endpoints

c64e5c8

UCP/TEST: Allow optional self RMA invalidation

7650e78

UCP/CORE: Preserve self failover RMA invalidation

7c04647

pentschev force-pushed the intra-process-cuda-copy-fix branch from e2aa013 to 7c04647 Compare August 25, 2026 10:13

Merge remote-tracking branch 'upstream/master' into intra-process-cud…

6e39794

…a-copy-fix

svc-nvidia-pr-review reviewed Sep 2, 2026

View reviewed changes

Comment thread test/gtest/ucp/test_ucp_peer_failure.cc Resolved

Comment thread test/gtest/ucp/test_ucp_peer_failure.cc Resolved

UCP/CORE: Clarify failover self handling

f1f8121

pentschev added 2 commits September 16, 2026 08:23

UCP/CORE: Remove unreachable self recovery handling

9b4825f

UCP/TEST: Strengthen self peer wireup coverage

f9e0c0f

svc-nvidia-pr-review reviewed Sep 16, 2026

View reviewed changes

UCP/WIREUP: Clarify same-worker peer handling

5f041ec

svc-nvidia-pr-review reviewed Sep 16, 2026

View reviewed changes

Comment thread src/ucp/core/ucp_request.c Resolved

Comment thread test/gtest/ucp/test_ucp_peer_failure.cc Resolved

UCP/CORE: Preserve invalidation on capable self lanes

00e37ec

svc-nvidia-pr-review reviewed Sep 28, 2026

View reviewed changes

Comment thread test/gtest/ucp/test_ucp_wireup.cc Resolved

Comment thread test/gtest/ucp/test_ucp_peer_failure.cc Resolved

UCP/TEST: Handle self endpoints without AM lanes

13b658e

svc-nvidia-pr-review reviewed Sep 28, 2026

View reviewed changes

UCP/TEST: Group peer wireup tests

85bf24e

svc-nvidia-pr-review reviewed Sep 29, 2026

View reviewed changes

rakhmets approved these changes Oct 5, 2026

View reviewed changes

gleon99 merged commit 694cf78 into openucx:master Oct 5, 2026
162 checks passed

vyasr mentioned this pull request Oct 7, 2026

fix: Deferred intra-node UCX exchange handshakes facebookincubator/velox#18921

Open

This was referenced Oct 11, 2026

TEST: Check request completion before blocking progress rapidsai/ucxx#777

Merged

AZP: UCXX integration - tests + builds #11473

Open

guy-ealey-morag mentioned this pull request Oct 12, 2026

GTEST: Restore and check the current CUDA device after each test #12073

Merged

svc-ucx mentioned this pull request Oct 14, 2026

UCT/IB/MLX5: support flush remote in purge outstanding #12062

Merged

-            /* Same-worker PEER selection may include MDs without RMA
+            if (!(ucp_ep_md_attr(ep, lane)->flags &
+                  UCT_MD_FLAG_INVALIDATE_RMA)) {
+                /* Same-worker PEER selection may pick MDs without RMA
+                 * invalidation */
+                ucs_assert(key->flags & UCP_EP_CONFIG_KEY_FLAG_SELF);
+                continue;
+            }

Conversation

pentschev commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

Why?

Uh oh!

Uh oh!

pentschev commented Apr 9, 2026

Uh oh!

yosefe left a comment

Choose a reason for hiding this comment

Uh oh!

yosefe Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

pentschev Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

svc-ucx commented Aug 24, 2026

Uh oh!

shasson5 commented Aug 24, 2026

Uh oh!

gleon99 commented Aug 25, 2026

Uh oh!

pentschev commented Aug 25, 2026

Uh oh!

pentschev commented Sep 2, 2026

Uh oh!

svc-nvidia-pr-review commented Sep 2, 2026

Uh oh!

Uh oh!

Uh oh!

svc-nvidia-pr-review commented Sep 2, 2026

Uh oh!

svc-nvidia-pr-review commented Sep 16, 2026

Uh oh!

svc-nvidia-pr-review commented Sep 16, 2026

Uh oh!

Uh oh!

svc-nvidia-pr-review Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

pentschev Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

svc-nvidia-pr-review commented Sep 16, 2026

Uh oh!

Uh oh!

Uh oh!

svc-ucx commented Sep 17, 2026

Uh oh!

svc-nvidia-pr-review commented Sep 28, 2026

Uh oh!

Uh oh!

Uh oh!

svc-ucx commented Sep 28, 2026

Uh oh!

gleon99 commented Sep 28, 2026

Uh oh!

svc-nvidia-pr-review commented Sep 28, 2026

Uh oh!

svc-ucx commented Sep 28, 2026

Uh oh!

svc-nvidia-pr-review Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

pentschev Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

svc-nvidia-pr-review commented Sep 29, 2026

Uh oh!

svc-nvidia-pr-review Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

pentschev commented Apr 1, 2026 •

edited

Loading

pentschev Aug 17, 2026 •

edited

Loading