[superleo]

What?

Add initial MUSA support to UCX .

The series includes:

• a stable UCS_MEMORY_TYPE_MUSA memory type and musa name;
• MUSA capability reporting through UCP context queries;
• UCM hooks for musaMalloc and musaFree;
• MUSA memory detection and registration in the UCT MUSA MD;
• MUSA host registration and DMA-BUF capability reporting;
• PCI BDF and UCX system-device topology mapping;
• the musa_copy staging transport for local MUSA/host transfers;
• IB peer-memory and DMA-BUF integration for direct GPU RDMA;
• default musa transport selection through musa_copy;
• build, configure, and focused contract tests.

The MUSA component no longer exposes a zero-capability stub TL. The copy path is provided by musa_copy, while large-message direct RDMA remains selected through the IB transports when available.

Why?

NIXL requires UCX to distinguish MUSA device allocations from host memory and to select a real transport path for MUSA buffers. Without this support, MUSA allocations can be classified as host memory, the wrong protocol can be
selected, or the worker can select a zero-capability MUSA interface and abort.

This series provides the minimum UCX-side support needed to identify MUSA memory and use either local staging or the existing IB direct-RDMA path.

How?

The implementation is intentionally limited to ordinary MUSA device allocations.

Supported paths:

• MUSA device memory detection;
• MUSA device and host memory registration;
• MUSA/host staging through musa_copy;
• IB peer-memory or DMA-BUF registration when the platform supports it;
• UCP PUT/GET combinations covered by the provider contract.

Managed memory, VMM allocations, IPC-specific paths, heterogeneous devices, and provider-specific direct-RDMA extensions are not claimed as complete features.

Test:

The MUSA test bed /S5000 demonstrated:

• single-node RMA matrix: 8/8 passed;
• two-node RMA matrix: 7/7 passed;
• D-D, H-D, D-H, and H-H PUT/GET coverage;
• cross-process and cross-node rkey exchange;
• direct-RDMA evidence from mt_peermem counters;
• DMA-BUF export evidence;
• byte-level validation and matching FNV-1a checksums.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (Codestyle AUTHORS file update check) · commit 7ef53c12

TL;DR: The "AUTHORS file check" failed because PR #12107's author (superleo <xlei.boll@gmail.com>) is not listed in the AUTHORS file; the fix is to run ./contrib/authors_update.sh locally and commit the resulting one-line AUTHORS update.

Full analysis

Summary: Codestyle stage "AUTHORS file check" exited with code 1 — git diff --exit-code detected an uncommitted modification to AUTHORS after contrib/authors_update.sh ran.

Root cause: The pipeline step computes base_revision=[REDACTED:Hex High Entropy String], head_revision=[REDACTED:Hex High Entropy String] and runs ./contrib/authors_update.sh <base>..<head>, then asserts the working tree is clean. The script found a commit author not present in AUTHORS:

Names:
++ superleo <xlei.boll@gmail.com>

and inserted superleo <xlei.boll@gmail.com> after line 150 (shasson5 ...), so git diff --exit-code reported the diff and returned 1. This is not a flake or infra issue — it's the intended guard (contrib/authors_update.sh:36-38 adds the entry when neither the email nor the name matches an existing line; AUTHORS:148-153 confirms no superleo/xlei.boll entry exists).

Implicated commit: [REDACTED:Hex High Entropy String] (PR #12107, branch feat/musa_support, author superleo <xlei.boll@gmail.com>) — a missing AUTHORS entry, not a regression in CI tooling.

File: AUTHORS:150 (insertion point) — check driven by contrib/authors_update.sh:18-46

Suggested fix: On the feat/musa_support branch, run:

./contrib/authors_update.sh <upstream-base>..HEAD
git add AUTHORS && git commit --amend --no-edit   # or a separate "AUTHORS: add superleo" commit

which adds superleo <xlei.boll@gmail.com> to the alphabetical lower-case block of AUTHORS, then force-push. Note the script matches names case-insensitively and by substring, so if the contributor is already listed under a different display name, instead make the commit author identity consistent with the existing AUTHORS entry (e.g. via git commit --amend --author=...) rather than adding a duplicate line.

Related: PR #12107 (openucx/ucx); no existing issue found for this failure signature — it is a per-PR onboarding step, expected for every first-time contributor.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id abcc4e10-a0e4-40d3-84d2-c6d5fd660c79 in the triage console for the audit trail.

[superleo]

hi @yosefe @shamisp @tonycurtis @brminich @hoopoepg This patch contains three logical areas: MUSA memory type/UCM integration, UCT MUSA and musa_copy support, and IB direct-RDMA/DMA-BUF integration.

Would you prefer to review this as one dependent commit series, or should we split it into three PRs before review?

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

This is ~1.8k added lines in a single PR; pls split it (e.g. memory type + ucm hooks, uct base/md, musa_copy transport + tests) so each part can be reviewed on its own.

How is the MUSA build validated? There is no buildlib/tools/builds.sh entry or CI job that configures --with-musa, so nothing compiles this code (ZE added a dedicated build mode for exactly this).

Mode: REQUEST_CHANGES

Residual coverage gaps:

• mem_buffer (test/gtest/common/mem_buffer.cc) has no MUSA support, so no UCT p2p (uct_p2p_rma) or UCP mem-type test ever exercises musa_copy put/get; the only new functional tests are the UCM hook test and a managed-memory rejection test.
• musa is not in UCT_MD_INSTANTIATE_TEST_CASE, so the generic test_md suite (reg/dereg, mem_query, dmabuf) does not run against the new MD.
• UCP_MEM_IS_GPU/UCP_DEVICE_MEM_TYPES in src/ucp/core/ucp_mm.h and the ucp_rndv_frag_default_* tables were not extended for UCS_MEMORY_TYPE_MUSA, so rendezvous pipeline selection treats MUSA as non-GPU memory — worth confirming that is intentional for this first version.

[superleo]

Tested provider capabilities

Memory-type and pointer contract

The integration test confirmed that:

• MUSA device allocations are recognized as MUSA memory rather than host memory when allocated after ucp_init/UCX memory hooks are installed;
• host pointers cannot be registered as VRAM;
• device pointers cannot be registered as DRAM;
• invalid device IDs and invalid device pointers are rejected;
• zero-length, overflowed and out-of-range descriptors are rejected;
• subranges and multiple descriptors are handled correctly;
• descriptor boundaries may be deliberately unaligned without corrupting adjacent ranges.

A critical implementation contract was identified: on a memtype-cache miss, the current UCX path assumes host memory instead of taking the MUSA detection slow path. Consequently, MUSA buffers must be allocated after the UCX context and its memory hooks exist. This is now enforced by the NIXL driver ordering and documented for integrators.

MUSA transport and data correctness

The provider passed 17/17 hardware scenarios covering:

• host-host, device-device, host-device and device-host;
• 1 B, 4 KiB, 64 KiB, 1 MiB and 64 MiB transfers;
• repeated transfer and registration cycles;
• eight and 64 descriptors;
• multiple visible devices and non-default device IDs;
• WRITE plus active-message notification;
• READ-back verification;
• ordered deregistration and remote metadata invalidation;
• caller and background progress modes;
• bounded peer-exit failure.

The transfer data uses non-zero patterns that encode peer, operation, descriptor index and offset. Both receiver-side notification validation and sender-side READ-back validation are byte-exact.

Progress and error handling

The caller-progress path was exercised without a backend progress thread. The target side periodically calls getNotifs() while waiting for the sender’s completion marker, and the request owner polls getXferStatus(). This demonstrates the intended UCX worker-progress contract when progress is owned by the caller.

Peer-exit testing produced a bounded failure rather than an infinite wait. The NIXL layer exposes this as NIXL_ERR_REMOTE_DISCONNECT; the existing explicit restore test validates the required invalidate-and-reload workflow.

Resource behavior

A 1,000-cycle register/transfer/release/deregister test completed with stable device memory and a process RSS plateau. The observed RSS plateau is attributed to UCX pools rather than an unbounded leak, but it is material for multi-process deployments and should be included in capacity planning.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

Suggested mode: REQUEST_CHANGES, empty review body (the two blockers are build breaks).

Checked and not reported: ucm_musamem_scan_regions_cb() prot/path condition looks inverted at first glance, but it matches cudamem.c/rocmmem.c exactly (regions with no access permissions are intentionally included) — not a bug.

Checked and not reported: Appending UCS_MEMORY_TYPE_MUSA before UCS_MEMORY_TYPE_LAST keeps all existing on-wire values (same as the gaudi addition); UCS_STATIC_ASSERT(UCS_MEMORY_TYPE_LAST <= 255) still holds.

Checked and not reported: Duplicate UCX_CHECK_MUSA/AM_CONDITIONAL([HAVE_MUSA]) expansion from both src/ucm/musa and src/uct/musa mirrors cuda.m4; mem_buffer.cc switches all have default:, so the new memory type does not break the gtest build.

Checked and not reported: Repeated uct_ib_check_gpudirect_driver() calls for MUSA are harmless (it returns early once the bit is set).

Residual coverage gap: mem_buffer still has no MUSA allocator, so uct_p2p_rma/UCP mem-type tests never exercise musa_copy put/get; musa is also absent from UCT_MD_INSTANTIATE_TEST_CASE, so generic test_md (reg/dereg, mem_query, dmabuf) does not run against the new MD.

Residual coverage gap: uct_musa_base_has_dmabuf() is compile-time only — there is no runtime/driver check and no DMABUF=try style config knob like cuda_copy, so dmabuf_mem_types may advertise support that always fails at runtime.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 0) · commit 171799b8

TL;DR: The ASan "roce on worker 0" job did not fail on any test or compile error — the Azure agent was shut down mid-build ("The agent has received a shutdown signal"), cancelling the ./contrib/test_jenkins.sh step while it was still compiling gtest objects; just re-run the pipeline.

Full analysis

Summary: Build 141575, job "AddressSanitizer roce on worker 0", was cancelled by the CI agent during the gtest compilation phase of ./contrib/test_jenkins.sh; no UCX test ever ran.

Root cause: Infrastructure, not code. The log shows uninterrupted compiler activity from 13:16:14 through 13:30:43 (largest gap between consecutive lines is only ~10 s, e.g. ucp/test_ucp_wireup.cc at 13:30:19 → ucp/test_ucp_request.cc at 13:30:23 → shutdown at 13:30:33), so the job was making steady progress and was not hung. At 13:30:33 Azure DevOps logged ##[error]The agent has received a shutdown signal. This can happen when the agent service is stopped, or a manually started agent is canceled., immediately followed by ##[error]The Operation will be canceled... and ##[error]The operation was canceled. The last application work in flight was g++ ... -o ucp/gtest-test_ucp_wakeup.o ucp/test_ucp_wakeup.cc — a normal compile, with zero warnings or errors anywhere in the captured log (-Werror builds were all succeeding, every mv -f ... .Tpo ... .Po completed cleanly). Typical triggers for this message are the agent VM being reclaimed/preempted/restarted, or the run being superseded (e.g. a new push to feat/musa_support auto-cancelling the previous PR run via a poll trigger).

Implicated commit: none — commit [REDACTED:Hex High Entropy String] (PR #12107) is not implicated; the failure is agent-side cancellation.

File: n/a (last compiled unit: test/gtest/ucp/test_ucp_wakeup.cc; step: contrib/test_jenkins.sh)

Suggested fix: Re-run the "AddressSanitizer roce on worker 0" job — no code change is warranted. If it reproduces on this PR, check agent-pool health/preemption for that worker rather than UCX code. Secondary observation (not the failure, but it widens the window for such cancellations): the ASan configuration spends ~14.5 minutes of the job purely compiling test/gtest objects at -O3 -g -fsanitize=address; enabling ccache for the ASan row or lowering optimization to -O1 for gtest sources would shorten the exposure and speed recovery.

Related: contrib/test_jenkins.sh recent changes 217e877 (Alexey Rivkin, AZP/TESTS: do not run test_rma_cuda/test_ucp_rkey_destroy) and 390a30e (Alexey Rivkin, AZP/TESTS: build once per GPU test row and share the tree with the workers) — reviewed and unrelated to this cancellation. No matching issue/PR found for the agent shutdown signature.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 5ea41712-a5c2-4fef-b0e5-bf4128967e39 in the triage console for the audit trail.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

Residual coverage gaps: (a) nothing exercises the dmabuf export path, so the page-alignment finding would not be caught by any test even on MUSA hardware; test_md_dmabuf.mem_query_dmabuf now runs for musa_cpy but only with a page-size allocation, which may accidentally satisfy the alignment requirement. (b) The iface-cleanup ucs_fatal path is only reachable at teardown with a failing runtime, which no test simulates. (c) Adding musa_copy to UCT_TEST_TLS (via UCT_TEST_GPU_MEM_TYPE_TLS) now instantiates every generic uct_test suite for the new transport; without a --with-musa CI job none of that newly-instantiated coverage actually runs.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

src/uct/musa/musa_copy/musa_copy_iface.c

device_rscs is allocated for every device, so this makes the current device every GPU on the node at iface close, even for entries that were never used — each musaSetDevice() creates a MUSA context and reserves device memory. pls skip a device_rsc whose short_stream and all queue_desc[][].stream are NULL and whose active_queue is empty; uct_musa_copy_event_desc_cleanup() already selects the device on its own.

src/uct/musa/musa_copy/musa_copy_iface.c

cuda_copy implements iface_estimate_perf and returns UCS_ERR_UNSUPPORTED for host→host so the transport is never picked for host-to-host copies. With uct_base_iface_estimate_perf here, can musa_copy be selected as a memtype lane for host→host and end up doing a plain memcpy() in uct_musa_base_copy_with_info()?

Residual coverage gap: nothing exercises the multi-device cleanup path — test_musa_copy* runs on a single musa_copy iface and the cleanup loop over all devices is only observable on a multi-GPU node, which no CI job covers.

Residual coverage gap: uct_musa_base_select_device()/uct_musa_base_copy() cross-device D2D is still untested, and uct_musa_copy_query_devices() caps itself to one iface, so a cross-GPU copy surfaces as an error rather than a fallback.

With musa_copy now in UCT_TEST_GPU_MEM_TYPE_TLS, every generic uct_test suite is instantiated for it, but there is still no --with-musa entry in buildlib/tools/builds.sh or contrib/test_jenkins.sh, so none of that coverage (nor the compile of src/uct/musa, src/ucm/musa, and the three new gtest files) actually runs anywhere.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

test/gtest/common/mem_buffer.cc

On the new is_async_supported() MUSA block: allocate()/release() abort with UCS_TEST_ABORT for async MUSA when HAVE_MUSA_MEM_POOL_ALLOC is not defined, so a caller that gates on this helper will fail instead of skip on an older MUSA SDK.

#if HAVE_MUSA && HAVE_MUSA_MEM_POOL_ALLOC
    if (is_musa_supported() && (mem_type == UCS_MEMORY_TYPE_MUSA)) {
        return true;
    }
#endif

config/m4/musa.m4

On the HAVE_MUSA_MEM_POOL_ALLOC link test (the "Optional async/pool allocation family" block): why are muMemAlloc_v2, muMemFree_v2, muMemAllocManaged, musaMallocPitch and muMemAllocPitch_v2 in the async/pool group? They are not async APIs, so a SDK missing only muMemAllocAsync/muMemFreeAsync silently drops the hooks for the plain driver _v2 alloc/free as well (musamem.c:49-63, 434-442), and then driver-API allocations are never reported to UCM. pls check them in a separate group.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (Tests new on worker 2) · commit e797e0d2

TL;DR: The build never ran any UCX code — the Azure DevOps Get sources task failed 15 times with fatal: could not read Username for 'https://github.com': terminal prompts disabled, i.e. the pipeline's GitHub credentials were not injected into the git fetch. Fix is to re-authorize/rotate the GitHub service connection (or the agent's credential helper) on swx-rdmz-ucx-new-01 and re-run; no code change is needed.

Full analysis

Summary: Job "new on worker 2" (Azure build 142132) failed in the checkout stage — git fetch of refs/pull/12107/merge (SHA [REDACTED:Hex High Entropy String]) returned exit code 128 on every one of the 5 retries × 3 inner attempts.

Root cause: Pure CI infrastructure / authentication failure, not a product or test defect. In the log the agent queries for the auth header and gets nothing back:

• git config --get-all http.https://github.com/openucx/ucx.extraheader → no output
• git config --get-all http.extraheader → no output
• git config --get-regexp .*extraheader → no output

It then runs git --config-env=http.extraheader=env_var_http.extraheader fetch ... origin --depth=100 +9f1dc143... and git immediately fails with fatal: could not read Username for 'https://github.com': terminal prompts disabled (20:21:56, 20:22:01, 20:22:10, … 20:24:11). Because the env-var-backed http.extraheader was empty/invalid, git fell back to interactive credential prompting, which is disabled on the agent, so the anonymous fetch of a PR merge ref was rejected. The secondary errors (fatal: ambiguous argument 'HEAD', "delete source folder instead") are just the retry logic cleaning an empty repo — symptoms, not causes.

Note the total failure window is ~2m20s of continuous retry activity with no large timestamp gaps, so this is not a hang or timeout. The job then skipped all build/test steps (Skipping step due to condition evaluation. Evaluating: SucceededNode() Result: False), which is why there is no compile or test output at all — the feat/musa_support changes in PR #12107 were never built.

Implicated commit: unknown — no repository commit is implicated; the failure occurred before checkout. The PR head [REDACTED:Hex High Entropy String] / merge ref [REDACTED:Hex High Entropy String] is incidental.

File: none (failing step is the Azure Pipelines Get sources / checkout task for openucx/ucx@refs/pull/12107/merge, agent swx-rdmz-ucx-new-01-07, workspace [REDACTED:Base64 High Entropy String])

Suggested fix:

1. Re-authorize the GitHub service connection used by the "UCX PR" pipeline — its token has most likely expired, been revoked, or lost repo scope. Rotate the credential (do not paste it into the PR/issue) and re-save the connection so Azure DevOps re-injects http.extraheader.
2. If the pipeline uses a GitHub App installation, confirm the installation still grants access to openucx/ucx; if it uses a PAT, confirm repo scope and non-expiry.
3. Check swx-rdmz-ucx-new-01-07 specifically — a stale/partially-configured agent (agent version 3.246.0) or a leftover ~/.gitconfig / credential-helper override on that host can suppress the injected header. Compare against a worker that checked out successfully in the same run.
4. As a workaround to unblock PR [MUSA]Feat: musa support #12107, re-queue the build (ideally pinned to a different worker). No change to the feat/musa_support branch will affect this outcome.
5. Optional hardening: have the checkout step fail fast and emit a clearer message when http.extraheader resolves empty, instead of burning ~2.5 minutes in 15 doomed retries.

Related: none — a repository search for this error signature returned only unrelated PRs (#12203, #12171, #11644, #11998, #12062), confirming this is an agent/credential issue rather than a tracked code bug.

🛡️ This comment had 2 potential secret(s) redacted (Base64 High Entropy String, Hex High Entropy String). See request_id 52a39486-4030-46d7-8eb8-8a9c4a85e3e6 in the triage console for the audit trail.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[superleo]

test/gtest/common/mem_buffer.cc

On the new is_async_supported() MUSA block: allocate()/release() abort with UCS_TEST_ABORT for async MUSA when HAVE_MUSA_MEM_POOL_ALLOC is not defined, so a caller that gates on this helper will fail instead of skip on an older MUSA SDK.

#if HAVE_MUSA && HAVE_MUSA_MEM_POOL_ALLOC
    if (is_musa_supported() && (mem_type == UCS_MEMORY_TYPE_MUSA)) {
        return true;
    }
#endif

config/m4/musa.m4

On the HAVE_MUSA_MEM_POOL_ALLOC link test (the "Optional async/pool allocation family" block): why are muMemAlloc_v2, muMemFree_v2, muMemAllocManaged, musaMallocPitch and muMemAllocPitch_v2 in the async/pool group? They are not async APIs, so a SDK missing only muMemAllocAsync/muMemFreeAsync silently drops the hooks for the plain driver _v2 alloc/free as well (musamem.c:49-63, 434-442), and then driver-API allocations are never reported to UCM. pls check them in a separate group.

fixed @svc-nvidia-pr-review

[svc-nvidia-pr-review]

Residual coverage gap: uct_musa_base_get_device_by_sys_dev() has no test; the multi-device sys_dev mapping is only exercised on a multi-GPU MUSA node, which no CI job covers (still no --with-musa entry in buildlib/tools/builds.sh or contrib/test_jenkins.sh, so none of the newly instantiated UCT_TEST_GPU_MEM_TYPE_TLS suites actually run anywhere).

Residual coverage gap: test_md.mem_type_detect_mds now asserts exact alloc_length/base_address for UCS_MEMORY_TYPE_MUSA, and test_md.mem_query requires sys_dev != UNKNOWN (via is_device_detected()), i.e. both depend on muMemGetAddressRange() returning the exact requested size and on the musaDeviceGetPCIBusId → topo mapping succeeding. Neither assumption is validated in CI.

[superleo]

Residual coverage gap: uct_musa_base_get_device_by_sys_dev() has no test; the multi-device sys_dev mapping is only exercised on a multi-GPU MUSA node, which no CI job covers (still no --with-musa entry in buildlib/tools/builds.sh or contrib/test_jenkins.sh, so none of the newly instantiated UCT_TEST_GPU_MEM_TYPE_TLS suites actually run anywhere).

Residual coverage gap: test_md.mem_type_detect_mds now asserts exact alloc_length/base_address for UCS_MEMORY_TYPE_MUSA, and test_md.mem_query requires sys_dev != UNKNOWN (via is_device_detected()), i.e. both depend on muMemGetAddressRange() returning the exact requested size and on the musaDeviceGetPCIBusId → topo mapping succeeding. Neither assumption is validated in CI.

@svc-nvidia-pr-review On CI: none of the hosted runners have the MUSA SDK or MUSA GPUs, so a --with-musa build entry would need runner/infra support from the project first; happy to add the entry once that exists. On the multi-device sys_dev mapping: it is exercised on an 8-GPU MUSA node -test_md.mem_query asserts sys_dev != UNKNOWN for every instantiated device, and the copy paths go through query_copy_pointer() -> uct_musa_base_get_device_by_sys_dev() per device.

On the test assumptions: with SDK 5.2.0, muMemGetAddressRange() returns the exact requested size and the PCI bus id -> topology mapping succeeds for all devices, so the exact-size assertion intentionally mirrors the CUDA branch of the same test; if a future MUSA driver rounds allocation sizes up like ROCm HSA does, the GE relaxation pattern for ROCm is right next to it. With the UNKNOWN guard in place, get_device_by_sys_dev() also has direct unit tests now, which run on any MUSA build even without GPUs attached.

[svc-nvidia-pr-review]

🚫 @superleo — not authorized to trigger a review on this repo. Contact a repo admin to be added to the allowed-users list.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

minor: cuda_hooks.cc skips under valgrind because bistro-patched driver functions may execute stale translations, and musamem.c installs bistro hooks on the MUSA driver API too.

Suggested change

	ucs::test::init();
	if (RUNNING_ON_VALGRIND) {
	UCS_TEST_SKIP_R("Valgrind may execute stale translations of "
	"Bistro-patched MUSA driver functions");
	}
	ucs::test::init();

[svc-nvidia-pr-review]

Residual gap: UCT_TEST_GPU_MEM_TYPE_TLS now instantiates every generic uct_test suite for musa_copy, but uct_musa_copy_estimate_perf() only rejects host→host and otherwise falls back to uct_base_iface_estimate_perf, so the blocking short path (musaMemcpyAsync + musaStreamSynchronize) is reported with send_post_overhead = 0; nothing checks that the advertised cost matches the blocking behavior.

Residual gap: uct_musa_md_ensure_context() relies on musaSetDevice() eagerly creating a context; only test_musa_copy_md.host_registration_on_fresh_thread touches this, and it cannot run without a --with-musa CI job.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (Tests new on worker 3) · commit 6d7b0179

TL;DR: The build failed because exactly one gtest out of 8896 failed — tcp/test_ucp_rma.put_nonblocking/2 <tcp,cuda_copy,rocm_copy/user_memh> — which on this worker only exercises the host→host UCP_MEM_MAP_NONBLOCK + user-provided-memh ucp_put_nbx path, so it hit the data-validation ADD_FAILURE in test_ucp_memheap::test_xfer; nothing in the MUSA diff is compiled in on this (no-GPU) worker, so this looks like a pre-existing flake in the non-blocking-map/user-memh path rather than a PR regression.

Full analysis

Summary: make test in ./contrib/test_jenkins.sh returned 1 (Makefile:5106: test, bash exit 2) because tcp/test_ucp_rma.put_nonblocking/2 (variant user_memh) failed; all other 8895 tests passed.

Root cause: Evidence from the log:

• [ FAILED ] tcp/test_ucp_rma.put_nonblocking/2, where GetParam() = tcp,cuda_copy,rocm_copy/user_memh — single failure, no crash, no SIGSEGV/abort, no watchdog/timeout, no Endpoint timeout/error-count noise, and continuous timestamped activity right up to tear-down (no hang; largest inter-line gap is 18.7 s for tcp/test_ucp_wireup_1sided.stress_connect2/2, which passed).
• The worker has no GPU: [ SKIP ] (CUDA memory is not supported). So test_mem_types() (test/gtest/ucp/test_ucp_rma.cc:145-164) only ran the host→host pair plus the final host→host UCP_MEM_MAP_NONBLOCK iteration.
• In that path the only failure-producing assertions are ADD_FAILURE() << "data validation failed" and the ASSERT_UCS_OK(ucp_mem_map/unmap) calls in test/gtest/ucp/test_ucp_memheap.cc:58-126 (failure text at line 115). Note that with user_memh the sender's buffer is mapped with the same mem_map_flags (test_ucp_memheap.cc:80-85), i.e. the last iteration maps the send buffer UCP_MEM_MAP_NONBLOCK and passes it via UCP_OP_ATTR_FIELD_MEMH — the variant-specific difference between the passing /0,/1 and the failing /2. Sizes/iteration counts in that loop are randomized (ucs::rand()), which fits an intermittent failure.
• The assertion text itself is not recoverable: the fetched log is the tail (~55 s) of a 3,184,827 ms run, and the failing test ran much earlier. Attribution to the PR is therefore not supported by the log: MUSA code is #if HAVE_MUSA-guarded (test/gtest/common/mem_buffer.cc:56-67, 142-151, 237-239) and inactive here, and the one unconditional change — UCS_MEMORY_TYPE_MUSA in src/ucs/memory/memory_type.h:49 — is consistently reflected in ucs_memory_type_names/ucs_memory_type_descs (src/ucs/memory/memory_type.c:28,44), so no enum/name-array skew.

Implicated commit: unknown — not attributable to a PR commit from the log. (Branch commits in this area: [REDACTED:Hex High Entropy String] / e91befa0 / 7515f80e, superleo — all MUSA-guarded and inactive on this worker.)

File: test/gtest/ucp/test_ucp_memheap.cc:115 (reached from test/gtest/ucp/test_ucp_rma.cc:283 via test_ucp_rma.cc:162-163)

Suggested fix:

1. Re-trigger the job. A single randomized-size failure with no other symptom is most likely a flake; if it does not reproduce, it is not a PR blocker.
2. If it reproduces, run it isolated to get the lost assertion text:
   GTEST_FILTER='tcp/test_ucp_rma.put_nonblocking/2' UCX_LOG_LEVEL=debug ./gtest (and repeat with GTEST_REPEAT), which will show whether it is data validation failed or a failing ucp_mem_map.
3. Make this failure diagnosable in CI regardless: the ADD_FAILURE() << "data validation failed" at test_ucp_memheap.cc:115 carries no context — append size, num_iters, send_mem_type/target_mem_type, mem_map_flags and user_memh, so a tail-truncated Azure log still identifies the failing iteration.
4. Investigate the specific combination exposed only by variant /2: a sender memh created with UCP_MEM_MAP_NONBLOCK and passed through UCP_OP_ATTR_FIELD_MEMH to ucp_put_nbx over TCP (lazy/deferred registration). Either verify UCP registers it on demand, or (if the combination is intentionally unsupported) stop propagating mem_map_flags to the user memh mapping in test_xfer and map it without UCP_MEM_MAP_NONBLOCK.

Related: PR #12107 ([MUSA]Feat: musa support); the dedicated coverage for this path is test_ucp_rma.put_nbx_nonblock_map_user_memh (test/gtest/ucp/test_ucp_rma.cc:324-361). No existing issue found for this test failing.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 03291ee5-ee1f-4ad6-af76-9790e63b765b in the triage console for the audit trail.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

src/uct/musa/base/musa_base.c

pls map musaErrorMemoryAllocation to UCS_ERR_NO_MEMORY in uct_musa_base_status(). uct_musa_md_mem_alloc() returns this status as is and uct_mem_alloc() propagates the MD status without falling back, so an out-of-memory musaMalloc() surfaces as UCS_ERR_IO_ERROR; cuda_copy, rocm_copy and ze_copy all return UCS_ERR_NO_MEMORY there, and callers (UCP fragment pool growth, test_md.alloc) treat only UCS_ERR_NO_MEMORY as a tolerable OOM.

    if (error == musaErrorInvalidValue) {
        return UCS_ERR_INVALID_PARAM;
    }

    if (error == musaErrorMemoryAllocation) {
        return UCS_ERR_NO_MEMORY;
    }

    return UCS_ERR_IO_ERROR;

test/gtest/ucm/musa_hooks.cc

The file-static event globals musa_alloc_event/musa_free_event keep only the last event and are never reset between tests, so an assertion such as EXPECT_EQ(ptr, musa_free_event.mem_type.address) can pass on an event left over from a previous test instead of on the event under test. Also, both the runtime and the driver symbols are hooked, so one musaMalloc() dispatches more than one MEM_TYPE_ALLOC event. can we collect events in a vector and search it, like cuda_hooks.cc does with check_mem_alloc_events()?

Residual coverage gap: nothing covers the error-status mapping of the MD (uct_musa_base_status), so the OOM-vs-IO-error distinction above is not exercised by any test; test_md.alloc would only expose it under real memory pressure on MUSA hardware.

Residual coverage gap: muMemAllocHost/muMemFreeHost are not in ucm_musa_driver_funcs (cudamem.c hooks cuMemFreeHost/cuMemFreeHost_v2), so host memory allocated and released through the MUSA driver API produces no MEM_TYPE_FREE event; musa_hooks.cc only covers the runtime musaFreeHost path. Worth confirming whether the MUSA SDK exposes those driver entry points.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (AddressSanitizer gpu on worker 3) · commit e386e243

TL;DR: The ASan gtest shard died because rc_verbs/uct_cancel_test.get_bcopy/1 <rc_verbs/mlx5_1:1> could never get TX resources for uct_ep_flush(UCT_FLUSH_FLAG_CANCEL) (200 s of UCS_ERR_NO_RESOURCE), then hung in the IB CQ-poll path for 15 min until the gtest watchdog aborted the process — a pre-existing IB/rc_verbs test hang on device mlx5_1, not something the MUSA PR touches.

Full analysis

Summary: make -C test/gtest test (shard 4/4, AddressSanitizer gpu worker 3) aborted with Aborted (core dumped) after rc_verbs/uct_cancel_test.get_bcopy/1 <rc_verbs/mlx5_1:1> failed and then hung.

Root cause: Timeline from the Azure log:

• 17:05:21 test starts on rc_verbs/mlx5_1:1.
• 17:08:41 (exactly 200 s later) test/gtest/uct/test_flush.cc:736: Failure — Error: No resources are available to initiate the operation. That is the ASSERT_UCS_OK_OR_INPROGRESS(status) after the 200-second retry loop in uct_cancel_test::flush_and_reconnect(): every uct_ep_flush(ep, UCT_FLUSH_FLAG_CANCEL, &done) returned UCS_ERR_NO_RESOURCE for the whole loop. uct_rc_ep_flush() (src/uct/ib/rc/base/rc_ep.c:610-625) returns NO_RESOURCE while uct_rc_iface_has_tx_resources() is false (no TX CQ credits / TX desc mpool empty) or txqp available <= 0 — i.e. none of the 100 outstanding get_bcopy RDMA READs posted by fill() ever completed, so no WQE/CQE slot ever freed for the cancel-flush.
• 17:08:41 → 17:23:41: a 15-minute gap with zero output — the process was hung, not slow. The 900 s gtest watchdog (test/gtest/common/test_helpers.cc:31,57) then fired: Connection timed out - abort testing, Caught signal 6. The backtrace shows the main thread inside libuct_ib.so → libmlx5.so → pthread_spin_lock, i.e. spinning in ibv_poll_cq/post path while the post-abort teardown (uct_cancel_test::cleanup() → uct_test::flush() → entity::progress()) waited for completions that never arrived on mlx5_1.

Nothing in this path is touched by the PR: the only PR-specific artifacts in the log are the new src/ucm/musa, src/uct/musa, src/tools/perf MUSA dirs, all of which report "Nothing to be done". The same shard ran rc_verbs fine on mlx5_0:1 (test_uct_loopback.envelope/0 passed); only the mlx5_1:1 instance stalls, which points at the second IB port/device on swx-rdmz-ucx-gpu-02 (link down / RDMA READ loopback not completing) rather than at the commit under test.

Implicated commit: unknown — not attributable to [REDACTED:Hex High Entropy String] (MUSA support); no recent commit touches uct_cancel_test or the rc_verbs flush path (last relevant: 1d9a1d4e "UCT/IB/MLX5: support flush remote in purge outstanding", mlx5-only, and b0d5d1a4 rc_ep cleanup — neither is on the rc_verbs get_bcopy cancel path).

File: test/gtest/uct/test_flush.cc:736 (loop at 718-745); resource decision in src/uct/ib/rc/base/rc_ep.c:610-625; watchdog in test/gtest/common/test_helpers.cc:31,57.

Suggested fix:

1. Re-run the job; this is an infra/flaky failure unrelated to the MUSA changes. Before re-running, check the health of mlx5_1 port 1 on swx-rdmz-ucx-gpu-02 (ibv_devinfo port state, ibstat) — a down/misconfigured second port makes the self-connected RDMA READs in uct_cancel_test never complete and reproduces this exactly.
2. Harden the test so a stuck transport fails fast instead of eating the 900 s watchdog: in flush_and_reconnect(), when the 200 s loop expires with UCS_ERR_NO_RESOURCE, abort the test (UCS_TEST_ABORT) and skip the unbounded cleanup work; also bound the unconditional while (done.count != 1) progress(); loop at line 743 and uct_test::entity::flush() (test/gtest/uct/uct_test.cc:1471-1478, while (status == UCS_INPROGRESS) with no deadline) with ucs::get_deadline().
3. If it reproduces on healthy hardware, investigate TX-descriptor/CQ-credit accounting for canceled get_bcopy ops in uct_rc_txqp_purge_outstanding() (src/uct/ib/rc/base/rc_ep.c:534), since a leaked tx.mp desc or CQ credit would keep uct_rc_iface_has_tx_resources() false forever.

Related: PR #12107 ([MUSA]Feat: musa support) — failure is not caused by it; no existing issue found for uct_cancel_test flush(CANCEL) NO_RESOURCE hangs.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 59277e48-b0ce-475a-80c8-0764a5224811 in the triage console for the audit trail.