[yafshar]

What?

Create the ze_copy command queues and command lists for copies to host memory and to another device on a dedicated copy engine queue group when the device has one, or else on any copy-capable group, instead of the default group 0. Copies within one device run on a queue group with both the compute and the copy flag, where the driver also places them. Move the queue group lookup used by ze_ipc into ze_base, so both transports share it.

Why?

ze_copy creates its queues and lists with a zeroed descriptor, so they use queue group 0. On the tested devices, group 0 is the compute group, so every ze_copy submission runs on the compute engine, alongside the application's kernels. Copies to a peer or to host should run on the copy engine, as the driver recommends. ze_ipc already does this. Copies within one device are different: the driver keeps its own on the compute engine, which can be faster for cache-resident same-device copies, so ze_copy keeps them there as well.

How?

• uct_ze_base_get_copy_ordinal() is the former uct_ze_ipc_find_copy_ordinal(), moved to ze_base. It returns a dedicated copy group (COPY without COMPUTE) when the device has one, otherwise any group that can copy. Compared to the ze_ipc version it queries the group count first and allocates that many entries instead of a fixed array of 16, accepts a NULL queue-count pointer, and takes the log level for its failure messages: error from ze_ipc and from ze_copy iface init, debug from the lazy per-device queue creation.
• uct_ze_base_get_compute_ordinal() returns the first group with both the COMPUTE and the COPY flag, since the queues on it run memory copies. Both lookups share one group query; the copy lookup keeps its dedicated-copy preference.
• uct_ze_copy_iface_get_queue() queries the allocation properties of both buffers. When source and destination are device memory of the same device, the copy runs on that device's compute queue, created on first use like the per-device copy queues of UCT/ZE/COPY: Run copies on the device which owns the buffer #12090; shared memory does not count, as it may reside on the host. Every other copy keeps UCT/ZE/COPY: Run copies on the device which owns the buffer #12090's rule: the copy queue of the device owning the source, else of the one owning the destination. Queue entries are keyed by device and engine; entry 0 stays the copy queue of the MD device. The single-device shortcut of UCT/ZE/COPY: Run copies on the device which owns the buffer #12090 goes away, as the same-device check needs the allocation properties anyway.
• The lookups run in uct_ze_copy_iface_queue_create(), the single place where UCT/ZE/COPY: Run copies on the device which owns the buffer #12090 creates queues, and set cq_desc.ordinal and cl_desc.commandQueueGroupOrdinal for the engine of the entry, so each list is submitted to the group of its queue. When a lookup or the queue creation fails on the lazy path, the copy keeps using the MD device queue, as in UCT/ZE/COPY: Run copies on the device which owns the buffer #12090. At iface init such a failure fails the iface; a device without a copy-capable group was not known to work on group 0 either.
• gtest: test_ze_copy_rma.queue_on_copy_engine first triggers the creation of a queue for a second device (when there is one), then for every queue of the iface reads the queue and list ordinals with zeCommandQueueGetOrdinal / zeCommandListGetOrdinal and checks that they match and that the group is the one the entry's engine asks for: a copy group, dedicated when the device has one, or a group with compute and copy. New same_device_copy_on_compute_engine checks that copies to and from host stay on the copy queue, that a copy between two device buffers selects a compute queue of that device in both directions, moves data through it with a put and with a get, and checks its group; shared_memory_copy_on_copy_engine checks that shared memory stays on the copy queue. queue_follows_buffer_device also moves data between two buffers of the second device through its compute queue and checks that queue's group. The ordinal queries are Level Zero API 1.9: configure checks for them (AC_CHECK_DECLS in ze.m4), and the tests skip when the headers do not declare them or the driver does not implement them. When the MD context does not resolve the other context's memory, the tests check the MD device queue alone.

Trade-off

Copies to a peer or to host are equal or faster on the copy engine and no longer compete with the application's kernels on the compute engine. Same-device copies keep sharing the compute engine with the kernels, as before this PR. On the tested devices the group selected for them is the group ze_copy used before, so their performance is unchanged; moving them to the copy engine would slow cache-resident same-device copies of a few MB. On one of the tested devices, small same-device copies have lower latency on the copy engine; choosing the engine per size would need device-specific thresholds and is not done here.

Testing

• Intel Arc Pro B60 and B70, two and four cards visible, with UCX_LOG_LEVEL=error: *ze_copy*:*ze_cpy*:*ze_ipc* gtests pass (67/67; 65/65 with the copy-engine commits alone, 64/64 on UCT/ZE/COPY: Run copies on the device which owns the buffer #12090 alone). Without that variable the same run has one failure, ze_cpy/test_md.sys_device/0, which fails identically on master: ze_copy warns when it limits two visible devices to one for the memtype endpoint, and gtest fails any test that logs a warning. The warning is unrelated to this change. The new tests report, on both cards, the same-device queue on a group with compute and copy and the copy queues on a dedicated copy group. Built against UCT/ZE/COPY: Run copies on the device which owns the buffer #12090 without this change, queue_on_copy_engine fails: the queues are on group 0 (COPY|COMPUTE).
• Engine placement checked with the driver's submission trace (PrintDeviceAndEngineIdOnSubmission): with UCT/ZE/COPY: Run copies on the device which owns the buffer #12090 alone, every ze_copy submission goes to the compute engine of the device running the copy; with this change, copies to host and between devices go to that device's copy engine and copies within one device to its compute engine.
• Copies issued the way ze_copy issues them, compared on both engines for every path (same card, card to card in both directions, host to and from device) and sizes from 8 B to 256 MiB, with payload validation: see the trade-off above. With the last commit, same-device copies match the numbers from before the copy-engine commits at every size on both devices, and every other path is unchanged.
• CUDA node (no Level Zero, so this code is not compiled): the tree builds with --without-gda; *cuda_copy*:*cuda_ipc* and the UCP memtype/proto gtests give the same results with and without this change.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

test/gtest/uct/ze/test_ze.cc

zeCommandQueueGetOrdinal()/zeCommandListGetOrdinal() require Level Zero API 1.9. config/m4/ze.m4 only checks for level_zero/ze_api.h and zeInit, and nothing else in the tree uses post-1.0 entry points, so gtest will fail to compile against older level-zero headers. There is no ZE build job in buildlib/contrib, so CI will not catch it.

zeCommandQueueGetOrdinal/zeCommandListGetOrdinal were added in Level Zero 1.9; pls guard them (ZE_API_VERSION_CURRENT) or add a configure check, otherwise gtest fails to build with older level-zero headers since ze.m4 only checks for zeInit.

src/uct/ze/copy/ze_copy_iface.c

minor: num_queues is only a dummy output here — it is unused by ze_copy; maybe let uct_ze_base_get_copy_ordinal accept NULL so the dummy variable is not needed.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[yafshar]

test/gtest/uct/ze/test_ze.cc

zeCommandQueueGetOrdinal()/zeCommandListGetOrdinal() require Level Zero API 1.9. config/m4/ze.m4 only checks for level_zero/ze_api.h and zeInit, and nothing else in the tree uses post-1.0 entry points, so gtest will fail to compile against older level-zero headers. There is no ZE build job in buildlib/contrib, so CI will not catch it.

zeCommandQueueGetOrdinal/zeCommandListGetOrdinal were added in Level Zero 1.9; pls guard them (ZE_API_VERSION_CURRENT) or add a configure check, otherwise gtest fails to build with older level-zero headers since ze.m4 only checks for zeInit.

Done. The header has no preprocessor-testable version macro before API 1.19, so ze.m4 now runs AC_CHECK_DECLS for zeCommandQueueGetOrdinal and zeCommandListGetOrdinal, and the test skips when they are missing. Debian 12 ships API 1.4 headers and Ubuntu 24.04 ships 1.9, so the guard is needed.

src/uct/ze/copy/ze_copy_iface.c

minor: num_queues is only a dummy output here — it is unused by ze_copy; maybe let uct_ze_base_get_copy_ordinal accept NULL so the dummy variable is not needed.

Done, num_queues_p may be NULL and ze_copy passes NULL.

[svc-nvidia-pr-review]

Both new tests need real Intel GPUs, and queue_follows_buffer_device additionally needs a second ZE device (otherwise it skips); there is no ZE job in buildlib/contrib, so CI will only compile this code, not execute it. The multi-device selection logic — which is the point of the PR — therefore has no automated coverage.

queue_on_copy_engine silently becomes a skip when built against Level Zero headers older than 1.9, so a regression in the ordinal selection would not be caught on older build environments either.

Residual behavior change worth the author's attention but not comment-worthy on its own: element 0 is now created on a copy-engine ordinal instead of the default group, so uct_ze_base_get_copy_ordinal() failing (device with no COPY-capable group) makes ze_copy iface creation fail where it previously succeeded on group 0. Intel GPUs report COMPUTE|COPY on the compute group, so the second pass should always match in practice.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 1) · commit a9227fb7

TL;DR: The ASAN gtest job failed because one test, ud/test_ucp_sockaddr_protocols_err.tag_rndv_unexp_get_scheme/17 (ud_v,cuda_copy,rocm_copy/bidi_stop,not_all_devs,mt), stalled for 678 s — two UCP requests never completed after the forced bidirectional EP close in the RNDV get_zcopy error flow, each hitting the gtest ~340 s request deadline. This is a hang in the UCP peer-failure/request-purge path, not a slow machine, and it is unrelated to the ZE copy-engine change in PR #12149.

Full analysis

Summary: make test in the "AddressSanitizer roce on worker 1" job exited with error 1: 8875/8876 tests passed, the single failure being ud/test_ucp_sockaddr_protocols_err.tag_rndv_unexp_get_scheme/17, which took 678083 ms (next slowest test: 17.6 s).

Root cause: The test runs test_tag_send_recv(64K, is_exp=false) with RNDV_THRESH=0, RNDV_SCHEME=get_zcopy and variant bidi_stop, i.e. both receiver and sender EPs are closed with UCP_EP_CLOSE_FLAG_FORCE while an unexpected RNDV GET-zcopy transfer is in flight (test/gtest/ucp/test_ucp_sockaddr.cc:3115 → :2336-2402). After the forced close, the in-flight send and tag-msg-recv requests were never completed — neither with UCS_OK nor purged with an error — so requests_wait() spun in ucp_test::request_process() until ucs::get_deadline() expired and logged "request %p did not complete on time" (test/gtest/ucp/ucp_test.cc:289-306). The base deadline is 180 s (test/gtest/common/test_helpers.cc:29) scaled by the ASAN time multiplier, which matches the observed ~339 s per stalled request × 2 requests ≈ 678 s total. It stayed under the 900 s watchdog (test_helpers.cc:31), so the process continued and only the one test was reported failed. Nothing in the failing configuration (ud_v,cuda_copy,rocm_copy) touches uct/ze/copy, so the PR's change is not implicated; the stall is in the UCP rendezvous GET-zcopy / lane-purge error flow.

Implicated commit: unknown — not the PR commit a9227fb (ZE copy engine, not exercised by ud_v). Prime suspects in the recently-changed RNDV/fault-tolerance purge path: [REDACTED:Hex High Entropy String] "UCP/PROTO: Repost GET and zcopy on the selected lane (#12091)", 40d8c4a6 "UCP/FT: Purge an unaware lane before rebuilding its address (#12048)" and 4f36f859 "UCP/CORE: Purge held outstanding work with the stored RX token (#12008)" (all Evgeny Leksikov, Oct 2026).

File: test/gtest/ucp/test_ucp_sockaddr.cc:3115 (test), stall observed at test/gtest/ucp/ucp_test.cc:289-306; suspected product code in src/ucp/core/ucp_ep.c / src/ucp/rndv/ request-purge-on-forced-close path.

Suggested fix: Re-run this job (and/or run the same test on current master) to confirm it is not PR-related — expected outcome is that it reproduces on master too, in which case the PR can proceed and a separate bug should be opened. For the real defect, instrument/inspect the forced-close purge path for RNDV GET-zcopy: when ucp_ep_close_nbx(FORCE) is called on both peers mid-rendezvous, all outstanding GET-zcopy and tag-msg-recv requests must be completed with UCS_ERR_CANCELED/UCS_ERR_ENDPOINT_TIMEOUT (ucp_ep_purge_lanes/ucp_request_complete on the selected GET lane). Running the single test with UCX_LOG_LEVEL=debug -gtest_filter='ud/test_ucp_sockaddr_protocols_err.tag_rndv_unexp_get_scheme/17' and dumping the request state at the deadline will pinpoint which request is left in INPROGRESS. Do not raise the gtest timeout — the log shows a stall, not slow progress.

Related: PR #12149 (this PR, UCT/ZE/COPY: Run copies on a copy engine); related purge-path PRs #12091, #12048, #12008. No existing issue found for this test.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 8f5cc60f-408c-4fdc-aea7-46c84218fac4 in the triage console for the audit trail.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

PR size exceeds the 500-line hard limit (~670 added lines) and mixes two independent changes. Please split: one PR moving find_copy_ordinal to uct_ze_base_get_copy_ordinal() (including the ze_copy switch to a copy-engine ordinal), and one adding the per-device queue array with its tests.

Residual coverage gaps: both new multi-device tests still require two real Intel GPUs and skip otherwise, and md_queue_when_create_fails fabricates the fallback entry by hand, so the queue_create failure path itself remains uncovered.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

test/gtest/uct/ze/test_ze.cc

The compile-time declaration check (ze.m4) does not imply runtime support: it only proves the installed header declares zeCommandQueueGetOrdinal/zeCommandListGetOrdinal. With a new SDK but an older loader/driver, the loader's DDI stub returns ZE_RESULT_ERROR_UNSUPPORTED_VERSION, so ASSERT_EQ(ZE_RESULT_SUCCESS, ...) turns a missing-feature environment into a test failure rather than a skip. Pls skip when the query is not supported.

Residual coverage gaps: both multi-device tests still require two resolvable ZE devices and skip otherwise; queue_on_copy_engine also silently skips when built against pre-1.9 headers, so the ordinal-selection logic has no coverage in those environments. Additionally, the queue_create() failure path itself remains untested (md_queue_when_create_fails fabricates the fallback entry by hand).

[yafshar]

PR size exceeds the 500-line hard limit (~670 added lines) and mixes two independent changes. Please split: one PR moving find_copy_ordinal to uct_ze_base_get_copy_ordinal() (including the ze_copy switch to a copy-engine ordinal), and one adding the per-device queue array with its tests.

Residual coverage gaps: both new multi-device tests still require two real Intel GPUs and skip otherwise, and md_queue_when_create_fails fabricates the fallback entry by hand, so the queue_create failure path itself remains uncovered.

This PR is stacked on #12090, as the first line of the description says: the per-device queue array and its tests are that PR, open for review. The change here is the last two commits, 217 added lines in 6 files. Once #12090 merges this branch is rebased onto master and the diff shrinks to those two commits.

[yafshar]

test/gtest/uct/ze/test_ze.cc

The compile-time declaration check (ze.m4) does not imply runtime support: it only proves the installed header declares zeCommandQueueGetOrdinal/zeCommandListGetOrdinal. With a new SDK but an older loader/driver, the loader's DDI stub returns ZE_RESULT_ERROR_UNSUPPORTED_VERSION, so ASSERT_EQ(ZE_RESULT_SUCCESS, ...) turns a missing-feature environment into a test failure rather than a skip. Pls skip when the query is not supported.

Residual coverage gaps: both multi-device tests still require two resolvable ZE devices and skip otherwise; queue_on_copy_engine also silently skips when built against pre-1.9 headers, so the ordinal-selection logic has no coverage in those environments. Additionally, the queue_create() failure path itself remains untested (md_queue_when_create_fails fabricates the fallback entry by hand).

Agreed, the declaration check covers the headers, not the loader. The next revision skips the test when the ordinal query returns ZE_RESULT_ERROR_UNSUPPORTED_VERSION or ZE_RESULT_ERROR_UNSUPPORTED_FEATURE. On the coverage note: correct, there is no Level Zero CI job; the tests ran on B60 and B70 as described, and a real queue creation failure cannot be provoked without fault injection in the driver.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 3) · commit afb6ad59

TL;DR: The ASAN roce job did not fail on an assertion — the gtest watchdog killed it after a 900-second hang inside rc/test_ucp_tag_match_rndv.req_exp/6 <rc_v/rndv_put_flush,proto_v1>; the hang is in the rc_verbs rendezvous PUT+flush completion path and is unrelated to this PR's Level-Zero-only change, so it should be re-run and tracked as a rndv_put/rc_verbs hang, not fixed by raising the timeout.

Full analysis

Summary: gtest aborted (SIGABRT via watchdog pthread_kill) because rc/test_ucp_tag_match_rndv.req_exp/6 <rc_v/rndv_put_flush,proto_v1> never completed; make test exited with "Aborted (core dumped)" → Bash exit code 2.

Root cause: Timestamp analysis shows a single, exactly-900-second gap in the log: the last application output is [ RUN ] rc/test_ucp_tag_match_rndv.req_exp/6 <rc_v/rndv_put_flush,proto_v1> at 22:28:33.406, and the next line is test_helpers.cc:57: Failure / Connection timed out - abort testing at 22:43:33.318. 900 s is exactly watchdog_timeout (test/gtest/common/test_helpers.cc:31), so the watchdog thread fired ADD_FAILURE() + pthread_kill(SIGTERM/SIGABRT). The backtrace confirms the test thread was spinning in the gtest progress loop (__sched_yield → gtest wait_for_flag/progress() frames), i.e. it was blocked waiting for a rendezvous transfer completion that never arrived — a hang, not slowness. The stuck configuration is rc_verbs with the rndv_put_flush variant, i.e. the rndv PUT protocol that relies on flush (instead of fence/ATP ordering) to signal remote completion (src/ucp/rndv/rndv_put.c, flush_map/atp_map logic, lines 22–31). Nothing in this PR can reach that code: commit afb6ad59 ("UCT/ZE/COPY: Keep copies within one device on the compute engine") touches only src/uct/ze/*, and the node's transport list in this run (shm, ib/rc_v/rc_x/ud, cuda_copy, rocm_copy, tcp, cma) contains no ZE device — every ZE/GPU test in the log is [ SKIP ]. The hang is therefore pre-existing/flaky in the rc_verbs rndv-put-flush path, exposed by ASAN's slowdown, not introduced by this change.

Implicated commit: Not this PR's commit ([REDACTED:Hex High Entropy String], Yaser Afshar, ZE-only). Most plausible prior suspects for the rndv PUT flush/fence ordering: 9a06403f "UCP/RNDV: Skip PUT fence for ordered transports (#11673)" and 4d0b5b18 "UCT/IB: Consume pending fence on send — revert UCT API changes (#11699)", both by Roie Danino.

File: Hang detected at test/gtest/common/test_helpers.cc:57 (watchdog); hung code under investigation: src/ucp/rndv/rndv_put.c:22-31 (flush/ATP completion for rndv_put_flush) exercised by rc/test_ucp_tag_match_rndv.req_exp in test/gtest/ucp/test_ucp_tag_match.cc.

Suggested fix: Re-trigger the roce ASAN job — this failure is not caused by the ZE change and should not block PR #12149. In parallel, open a separate bug for the hang: reproduce with ./gtest --gtest_filter='rc/test_ucp_tag_match_rndv.req_exp/6' --gtest_repeat=50 with UCX_LOG_LEVEL=trace on rc_verbs + rndv_put_flush, and inspect whether the rndv PUT completion is lost when the fence is skipped for ordered transports (ucp_proto_rndv_put_common_flush* / ATP path). Do not raise watchdog_timeout — the process made zero progress for the full 900 s, so a larger timeout would only delay the same abort.

Related: PR #12149 (this build). No existing issue found for test_ucp_tag_match_rndv.req_exp + rndv_put_flush hangs; candidate regressions to review: #11673, #11699.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 96b8b14c-e15a-4288-a959-f608450ec25d in the triage console for the audit trail.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

PR size is still ~670 added lines and mixes the get_copy_ordinal() move with the per-device queue array; the split request from the earlier review is unresolved (support the existing thread rather than re-raising).

Coverage: both multi-device tests need two resolvable ZE devices and skip otherwise, queue_on_copy_engine/check_queue_engine skip when built against pre-1.9 headers or when the driver does not implement the ordinal query, and the real queue_create() failure path is still only simulated by hand-inserting the fallback entry. There is no ZE job in buildlib/contrib, so CI compiles but never runs this logic.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (Tests gpu on worker 3) · commit 4f533d2f

TL;DR: The gpu-worker-3 gtest run failed on a single unrelated test, dc_mlx5/uct_p2p_err_test.local_access_error/1 (loopback), which races on an asynchronously-delivered IB local-protection error completion; PR #12149 only touches Level-Zero (src/uct/ze/copy) code that isn't exercised on this NVIDIA/mlx5 worker, so retrigger the job and de-flake the test's fixed 10 ms wait.

Full analysis

Summary: make test exited 1 because 1 of 10533 gtests failed: dc_mlx5/uct_p2p_err_test.local_access_error/1, where GetParam() = dc_mlx5/mlx5_0:1/loopback (all other 10512 tests passed, 20 skipped).

Root cause: Not a hang or timeout — the log shows continuous activity to the end (largest gap ~17 s inside the legitimately slow test_ucp_atomic*.fetch tests) and the suite ran to completion before make: *** [Makefile:4972: test] Error 1.

The failing test posts a uct_ep_put_zcopy from a deliberately unmapped local VA (get_unused_address()) with a valid memh and expects the transport to report a local access error. In uct_p2p_err_test::test_error_run() the call returns UCS_INPROGRESS (the error is only discovered later, when the NIC generates the error CQE), so the test falls into the else branch and waits only wait_for_remote() + a hard-coded ucs::safe_usleep(1e4) (10 ms) before counting captured error messages. Because the pattern passed from local_access_error is the empty string, every captured message matches and num_unmatched can never be non-zero — so the only assertion that can fire is EXPECT_GT(num_matched, 0ul) ("No error which contains the string '' has occurred during the test"), i.e. the error CQE / error_handler log had not been delivered yet within the 10 ms window. This is aggravated in loopback mode on DC, where (per the comment at test_p2p_err.cc:144-148) the IB TL may produce a QP flush error instead of/before the remote access error, making error delivery timing non-deterministic. On a loaded CI worker this is a classic timing flake.

Note: the Azure console log is truncated to the last ~500 lines, so the test's own assertion text was not retrievable — the above is derived from the test source and the fact that only EXPECT_GT(num_matched, 0) is reachable with an empty pattern.

Relevance to this PR: the branch fix/ze-copy-copy-engine (PR #12149, "UCT/ZE/COPY: Run copies on a copy engine") only changes src/uct/ze/copy/*, the Intel Level-Zero transport. The failing test is an IB dc_mlx5 loopback test with no ZE involvement, and test/gtest/uct/test_p2p_err.cc has not changed since 2022 (bca0bf5b), nor has src/uct/ib/dc/dc_mlx5.c recently. There is no plausible causal path from the PR to this failure.

Implicated commit: none — the PR commit 4f533d2f (ZE copy engine) is not implicated; the flaky test's last functional change is 146606ae (Alina Sklarevich, "UCT/UCP/GTEST: lower the log level for handled network errors")

File: test/gtest/uct/test_p2p_err.cc:103 (fixed 10 ms wait) and test/gtest/uct/test_p2p_err.cc:118 (the failing EXPECT_GT(num_matched, 0ul)), test case at test/gtest/uct/test_p2p_err.cc:157-169

Suggested fix:

1. Immediate: re-run the "gpu on worker 3" job — this failure is unrelated to PR UCT/ZE/COPY: Run copies on a copy engine #12149 and should not block it.
2. Real fix (separate PR): make test_error_run() wait for the error instead of sleeping a fixed 10 ms. Replace

   wait_for_remote();
   ucs::safe_usleep(1e4);

   with a bounded progress loop, e.g. wait_for_value/wait_for_flag-style polling that calls progress() until !m_errors.empty() || last_error != UCS_OK, with a deadline of ucs::test_time_multiplier() seconds, then assert. That removes the CI-load sensitivity while keeping the test meaningful.
3. Optionally scope the loopback leniency: since local_access_error/remote_access_error pass an empty error_pattern, num_unmatched is dead code there; passing a real pattern (e.g. "local" / the expected status string) would make the test assert something stronger and fail more informatively.

Related: PR #12149 (#12149) — the PR under test; related ZE work in #12090. No existing issue found tracking uct_p2p_err_test.local_access_error flakiness — worth opening one.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

Residual coverage gap (unchanged, no new comment): there is still no ZE job in buildlib/contrib, so the device-selection logic is compiled but never executed in CI.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

this adds 1-2 zeMemGetAllocProperties() calls per operation on the data path, also for put_short/get_short. can we cache the address→queue result instead of querying the driver per copy (uct_cuda_copy_get_mem_type() goes through ucs_memtype_cache_lookup() first, and ze_copy_md already updates that cache)?

[svc-ucx]

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 1) · commit 73a1e859

TL;DR: The ASAN RoCE gtest run failed on a single test, rc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64/1 (8895/8896 passed), a multi-threaded IB atomic fetch-add stress test that has no code path touching the PR's Intel Level-Zero copy changes — re-run the job; treat it as unrelated flakiness, not a regression from #12149.

Full analysis

Summary: make test in the "AddressSanitizer roce on worker 1" job exited non-zero because exactly one gtest failed: rc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64/1, where GetParam() = rc_mlx5/mlx5_1:1.

Root cause: Evidence from the log:

• [==========] 8896 tests from 1273 test suites ran. (2497035 ms total) / [ PASSED ] 8895 tests. / [ FAILED ] 1 test, then make: *** [Makefile:4971: test] Error 1 → ##[error]Bash exited with code '2'.
• No hang: timestamps run continuously from 00:41:27 to 00:41:53 with sub-second gaps right up to the tear-down, so this is a genuine assertion failure, not a wall-clock kill.
• The failing test is an uct_amo_test-family fetch-and-add test: num_senders() (4) threads each issue count() (1000) uct_ep_atomic64_fetch(UCT_ATOMIC_OP_ADD) operations against one 64-bit location, then validate_replies() asserts every expected reply value appears exactly once (UCS_TEST_ABORT("Reply ... appeared ... times; expected: ...")) and EXPECT_EQ(value, *(T*)recvbuf.ptr()) checks the final sum (test/gtest/uct/test_amo_fadd_fxor.cc:36-39, test/gtest/uct/test_amo.cc:66-116). This is a timing/concurrency-sensitive HW-atomics stress test, exactly the class of test that goes flaky under AddressSanitizer's thread slowdown on a loaded shared worker.
• The PR under test, UCT/ZE/COPY: Run copies on a copy engine #12149 "UCT/ZE/COPY: Run copies on a copy engine", only touches src/uct/ze/copy (Intel Level Zero). That transport is not present in this ubuntu2204_ib_...mofed570230 container — the log shows only cuda_copy/rocm_copy memtype variants, with CUDA/ROCm paths being skipped ([ SKIP ] (CUDA async allocation is not supported)). There is no mechanism by which a ZE copy-engine change affects an rc_mlx5 64-bit atomic fetch-add result.

Caveat: the fetched log is truncated to the last ~500 lines, so the actual assertion text for fadd64/1 (emitted ~2400 s earlier in the run) is not visible. The diagnosis of "unrelated flake" rests on the pass/fail ratio, the test's nature, and the PR's file scope rather than on the assertion message itself.

Implicated commit: unknown — no commit in the PR ([REDACTED:Hex High Entropy String], branch fix/ze-copy-copy-engine) touches IB atomics; the ZE copy module's last changes are unrelated (e.g. 272c989, Yaser Afshar, src/uct/ze/copy/ze_copy_md.c).

File: test/gtest/uct/test_amo_fadd_fxor.cc:48 (fadd64), assertion sites test/gtest/uct/test_amo.cc:106 and test/gtest/uct/test_amo_fadd_fxor.cc:39

Suggested fix: Re-trigger the "AddressSanitizer roce on worker 1" job — this is a single flaky HW-atomics stress test unrelated to the ZE change, and should not block #12149. If it reproduces, get the full (untruncated) job log to capture the assertion text, which distinguishes the two failure modes: a Reply ... appeared N times; expected M abort (lost/duplicated atomic reply — investigate rc_mlx5 atomic completion handling / IB_TX_INLINE_RESP path) versus the EXPECT_EQ(value, *recvbuf.ptr()) mismatch (final sum wrong — investigate the atomic mkey registration used by this key_reg_rdma_mem_type variant). As a CI hardening measure, consider scaling uct_amo_test::count()/num_senders() down under AddressSanitizer the same way it is already reduced under Valgrind (test/gtest/uct/test_amo.h:22-27 keys only off RUNNING_ON_VALGRIND), since ASAN similarly perturbs the multi-threaded timing this test depends on.

Related: PR #12149 (the PR under test, #12149); no existing issue found tracking uct_atomic_key_reg_rdma_mem_type.fadd64 flakiness — worth opening one to track it.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 8048d412-7c5f-457f-b561-880ef0b9da11 in the triage console for the audit trail.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

With the single-device fast path gone, every put/get now does up to two zeMemGetAllocProperties() calls, including on single-GPU nodes where get_queue() used to return immediately with zero driver calls. The dst lookup is also needed for the same-device/compute-engine decision, so it can't be avoided by reordering. Can we resolve this more cheaply, e.g. cache the last resolved pointer -> (device, is_device) in the iface, similar to how uct_cuda_copy_get_mem_type() tries ucs_memtype_cache_lookup() before falling back to a driver query (uct_cuda_copy_md_mem_query())?